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ABSTRACT 



This report is the result of a seven-month Urban 
Institute analysis of ESEA Title I evaluation and technical 
assistance problems and prospects. The focus of the study was on 
improvement of Federal and State evaluation and technical assistance 
related to education of disadvantaged children. During the study, 
site visits were made to five representative State Departments of 
Education. Chapter 1 is an overview of the major findings, 
conclusions, and recommendations of this report. The next three 
chapters are concerned with evaluation of education programs for 
disadvantaged children. Chapter 2 is a discussion of Federal Title I 
evaluation: priorities, present status, next steps. Chapter 3 focuses 
on State evaluation, and Chapter 4, on local evaluation. Chapter 5 
examines the implication of existing methodology for evaluation of 
the Title I program and discusses requirements for further instrument 
development. Chapter 6 provides an overview of technical assistance 
for programs for disadvantaged children; Chapter 7, Federal technical 
assistance; Chapter 8, State technical assistance efforts; and. 
Chapter 9, the local role in technical assistance to education 
programs for disadvantaged children. (Author/JM) 
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PREFACE 



This report is the result of a seven-month Urban Institute analysis 
of Title I evaluation and technical assistance problems and prospects. 

The Urban Institute conducted the work in response to requests from the 
Commissioner^ Title I [Elementary and Secondary Education Program] Task 
Force and from the Office of Program Planning and Evaluation in the U. S. 
Office of Education. The Urban Instituted work was performed under 
Office of Education contract 0EC-0-70-331 1 , dated March 16, 1970. 

The focus of the study was on improvement of Federal and State 
evaluation and technical assistance related to education of disadvantaged 
children. During the study, site visits were made to five representative 
State Departments of Education--and reports on those site visits have been 
made available to the Office of Education. 

Chapter I is an overview of the major findings, conclusions, and 
recommendations of this report. The next three chapters are concerned 
with evaluation of education programs for disadvantaged children. 

Chapter II is a discussion of Federal Title I evaluation: priorities, 

present status, next steps; Chapter III focuses on State evaluation; 
Chapter IV, on local evaluation. Chapter V examines the implication of 
existing methodology for evaluation of the Title I program and discusses 
requirements for further instrument development. 

Chapter VI provides an overview of technical assistance for programs 
for disadvantaged children; Chapter VII, Federal technical assistance; 
Chapter VIII, State technical assistance efforts; and Chapter IX, the 
local role in technical assistance to education programs for disadvantaged 
children. 

Available in a separate volume is a set of appendices containing 
information on a variety of topics related to Title I evaluation and tech- 
nical assistance. 
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I. OVERVIEW 



Although evaluation is a mandated activity at every administrative 
level of the Title I Elementary and Secondary Education program, there is 
little evidence that past evaluations have had much impact on the opera- 
tions of the program. Some critics claim that evaluation has had no impact 
because of inability to measure and record changes in so diverse a program: 
what should evaluators measure in a program in which objectives and treat- 
ments vary from town to town or State to State? Others attribute the failure 
of evaluation to a lack of adequate methodology: the evaluator lacks 

adequate tools to measure various aspects of educational growth, much less 
to account for any changes observed. Still others claim that evaluation 
results have had little impact because the basic notion of evaluation is a 
threat to the institution being evaluated. 

Yet 5 despite the problems, evaluation should be a vital part of the 
$1.2 billion per year Title I program, and the role of evaluation in 
improving program operations should be strengthened. It is through eval- 
uation that program administrators, parents, teachers, and other interested 
parties can estimate what impact Title I projects are having on children, 
teachers, and school systems, and can identify weaknesses in project 
operations. Using this information, appropriate technical assistance can 
be provided to improve educational services for disadvantaged children. 

The impact of evaluation is directly related to its ability to generate 
timely and useful information. This report reviews current Federal and 
State (and to a much lesser extent, local) efforts to evaluate the Title I 
program and to provide technical assistance in the improvement of program 
operations. It establishes a framework for Title I evaluation, by differ- 
entiating types of evaluation and attempting to identify the relative impor- 
tance of each type of evaluation for various decisions and decision-makers. 

This study finds that, regardless of the decision or the level of 
decision-making, program reporting, periodic needs assessment, and monitor- 
ing (assessment of the managerial and operational efficiency of programs 
and projects) emerge as the most needed Title I evaluation activities. 

Before the most basic decision can be made (or before more sophisticated 
evaluation can take place) , information must be available about the popula- 
tions served, services provided, the costs of providing services, and the 
relationships among current measures of educational need and contextual 
variables. Similarly, decision-makers at every level need to have some 
management assessment of the operations of projects and programs. Project 
evaluation/project rating and program strategy evaluation come next in 
importance for decisions concerning the allocation of funds and services 
for disadvantaged children. These types of evaluation focus on relative 
effectiveness of different local projects or different program components-- 
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information which is critical to decisions about the most effective alloca- 
tion of services within the Title I program. Program impact evaluation, 
which is concerned with the effectiveness of an entire national program, is 
important only for global decisions on total levels of funding. National 
program impact evaluation in the Title I program itself is of relatively 
little use either to local or to State education decision-makers. 

With regard specifically to Federal evaluation efforts, despite 
increases in the past few years in the funds available for evaluation, 
these resources are still in short supply. Much less than 1% of the Office 
of Education’s budget is used in support of its evaluation activities. 

Even if the available funds doubled or tripled, however, evaluation efforts 
would still be hampered by shortages of personnel, by limitations in exist- 
ing methodology, and by the availability and usability of data on many 
important aspects of education. 

Given all of these limiting conditions, it is important to determine 
how best to use available Federal evaluation funds to generate useful 
information for decision-makers. The priority assigned to each type of 
evaluation depends on several factors: (a) how often particular types of 

decisions are made, (b) how long it takes to carry out the particular type 
of evaluation, and (c) whether the evaluation is methodologically feasible. 
One fact about evaluation is often overlooked: the time required to mount 

and carry out an evaluation and the period of the decision-making cycle 
often fail to coincide. Evaluation can be a time-consuming process, with 
some types of evaluation requiring two or more years to plan and carry out; 
key educational decisions are often made annually. The inability of eval- 
uation to respond rapidly to the needs of decision-makers makes more 
difficult attempts to build support for the process of evaluation, 

In formulating a long-term strategy for Federal Title I evaluation 
efforts, this study finds that Congressionally-mandated evaluations of the 
national impact of the Title I program, of alternative Title I program 
components, and of individual local Title I projects are all very difficult 
methodologically, because of the low level of Federal inputs relative to 
local inputs and because of the absence of suitable comparison groups. The 
Office of Education should give much higher priority to Federal and State 
monitoring of Title I operations and comparative rating of the relative 
outputs of Title I projects with similar objectives. Congressional demands 
for Title I program reporting and needs assessment should be met but should 
probably be done on a two-year cycle (since observable changes will occur 
relatively slowly), in order to free resources for Title I monitoring and 
project rating. Monitoring and project rating need to be done on an annual 
cycle, to assist decisions on project refunding and on allocation of tech- 
nical assistance to individual projects. 

The Office of Education should act to shift Title I evaluation resources 
to systematic project classification and monitoring and to project rating, 
both to allow more effective (State) management of the Title I program and 
to aid in detecting promising projects that could later be subjected to more 
rigorous program strategy evaluation and then given wide dissemination. 

Title I evaluation should be directed increasingly toward systematic rating 
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of the relative outputs of different projects with similar objectives and 
serving similar types of children, to identify which projects appear to 
work best in which circumstances. (Such comparisons might be made 
nationally, regionally, or within individual States.) The Office of 
Education should decide which types of projects are of greatest interest 
(for example, reading projects, early childhood education projects, dropout 
prevention projects) and then focus Federal evaluation resources and induce 
States and localities to focus their own evaluation resources on project 
rating evaluation designed to pick out the projects that appear to be doing 
the best job within each of these project categories. 

At present, between $2 million and $4 million per year is available 
for Federal Title I evaluation, but relatively little effort is going into 
project rating or into improving Title I monitoring. Title I monitoring 
efforts are quite def icient--hence fine-sounding Title I regulations and 
guidelines have often been ignored in practice. The largest Federal 
evaluation expenditures are being made in support of the Belmont system, 
a cooperative OE-State evaluation effort begun in late 1968. Development 
of the Belmont system has been guided by two factors: the desire for 

cooperation between the Office of Education and the States and the need to 
consolidate existing evaluation and statistical reporting requirements. 

Thus far, Belmont has concentrated its efforts on the development of 
program reporting and needs assessment, although current efforts include 
the development of output measures and initial steps toward gathering data 
for project classification (through the Project Descriptor Instrument). 

Based on a review of the various components--both operational and planned-- 
of the Belmont system, it appears that Belmont is a promising effort at 
cooperative Federal-State data collection and evaluation, which is poten- 
tially very useful for Title I evaluation. The States are pleased with their 
partnership role in Belmont. Whenever possible, therefore, new cooperative 
Title I evaluation efforts should be launched under the Belmont umbrella. 

Belmont has, however, developed a number of data collection instruments 
without having a clear data analysis plan and plan for use of the data. 
Belmont looks like a "big system," in danger of being designed relatively 
independently of potential users of the system, not furnishing enough short- 
term help to motivate potential users to invest the staff time required to 
make the system relevant and, therefore, in danger of being a waste of money. 
Belmont was started in the right direction. With sufficient staff for 
Belmont and liaison staff from the Division of Compensatory Education and 
from State and large city agencies, Belmont could still become a model of 
cooperative development of data collection and evaluation systems of use 
to large numbers of Federal, State, and local personnel. 

Like the Federal government. State and Local Educational Agencies share 
responsibility for Title I evaluation. Title I funds are allocated accord- 
ing to a predetermined formula for use by local school districts to support 
a wide range of services for disadvantaged children. State Departments of 
Education play a pivotal role in the administration of the Title I program, 
since the States must approval all local project applications and have 
responsibility for monitoring local operations to see that they conform 
both with the letter and the spirit of the law. State evaluation efforts. 
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like Federal and local efforts, should be directed at improving the effec- 
tiveness of education programs and at improving the efficiency of the 
mechanisms which are involved in the delivery of those programs. 

i 

Evaluation is a relatively new phenomenon at the State level. In many 
cases, the impetus for evaluation came exclusively from Federal requirements 
and specifically, from the requirements of the Elementary and Secondary 
Education Act. If the sample of States visited during this study is in the 
least representative of activities among all 50 States, then it is quite 
clear that the Federal requirements continue to shape and to dominate State 
evaluation activities. Unfortunately, while States expend time and effort 
to meet Office of Education evaluation requirements, the information so 
generated is rarely useful to the State and only slightly more useful to the 
Office of Education. 

Since FY 1968, the Office of Education has carried out a national 
survey of selected elementary schools to gather information on compensatory 
education services and programs. This survey is now part of the Belmont 
system. In order to supplement the information in the survey, the Office 
of Education has revised the rather detailed set of questions previously 
used in the preparation of State annual Title I evaluation reports. Since 
FY 1968, the requirements for State annual reports have taken the form of 
between 9 and 13 open-ended questions about Title I activities. These 
broadly framed questions were intended to remove some of the more burdensome * 

requirements on the States and to allow the States the flexibility to develop 
evaluation systems more nearly suited to State needs. This does not appear 
to have happened to any significant degree. 

Typically, States send out evaluation guidelines or requests for infor- 
mation to Local Educational Agencies long after the school year had begun. 

Requests go out in the spring or near the end of the academic year, when the 
State is in a very poor position to influence either the evaluation design 
or the kind of information maintained on Title I projects. As a result, the 
forms collect ex post facto information which should be readily available to 
any project manager. The exception to this general statement is information 
collected on student achievement. While most States merely collect whatever 
test data exist (as opposed to requiring uniform testing or the use of 
selected tests) , the test data are often not processed until late in the 
summer. The reports filed by the Local Educational Agencies therefore reach 
the State Departments after decisions on project approval for the next year 
are made. 

The consequences of the timing and the content of State efforts to meet 
Federal requirements are twofold: (1) the extent to which the State affects 

local evaluation efforts is minimized, and (2) the utility to the State 
decision-making process of the information collected is greatly reduced. As 
presently constructed, the annual State reports provide little information v 

on the extent to which projects are meeting stated objectives. In all too 
few instances do Title I project evaluation designs include comparison or 
control groups. The information reported on achievement, while accounting 
for a significantly greater number of children than does the national survey, 
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contains nothing to indicate the characteristics of the projects for which 
data are reported or the characteristics of the pupils involved. States 
are asked to relate project costs to benefits, but the validity and relia- 
bility of the responses--where any are provided--are in question. 

What remains as the keystone of State evaluation activities is the 
variety of activities which we have called "monitoring . " Monitoring 
activities vary in content, structure, organization, and emphases across 
the States. No matter how haphazardly monitoring activities are carried 
out, they provide the direct opportunity for the State to influence local 
project development and to collect information for use by State personnel in 
decisions about project funding. Models for effective and efficient State 
program monitoring remain to be developed. Such models will have to take 
into account the variation that exists among the States in the way evalua- 
tion and monitoring activities are organized, the staff involved in such 
activities, and the resources available for their support. 

Because of the strategic role played by State Departments of Education 
in approving local Title I project applications, the Office of Education 
should give the highest Title I evaluation priority to improving the State 
monitoring activities through the development of models for State monitoring. 
Not one of the States visited during this study had a comprehensive monitor- 
ing system. For example, none could ensure that deficiencies found during 
site visits were corrected. Through a cooperative effort between the Office 
of Education and several States, several different models for monitoring 
systems could be developed. The models might contain a common core of 
information, but could allow for variations in State organization and in 
the composition of student populations in each State. Similarly, the Office 
of Education should develop a more systematic approach to its own monitoring 
of State Departments of Education. There should be a restructuring of the 
format of the annual State Title I reports to focus primarily on reporting 
the results of State monitoring activities. This would greatly enhance the 
value of the annual State report, both to the Federal government and to the 
State involved in the report's preparation. 

In the. area of local evaluation, the Office of Education should work 
with a number of States to develop and test cooperative local evaluation 
efforts, which would include the use of common evaluation designs and common 
output measures. Local project evaluation is an integral part of the 
Title I program, but its utility for decision-making is greatly diminished 
because the results of these evaluations are neither comparable nor timely. 
Although the precise amount being spent on local Title I project evaluation 
is not known, an estimate of $10 million per year is not unreasonable. 

Most local evaluations are undertaken to fulfill the mandated requirements 
of ESEA and are often not completed in time to be of use in local decision- 
making. For local decision-making, evaluations of individual Title I 
projects should focus on relatively inexpensive short-term comparisons of 
project results with previously stated performance objectives or with 
baseline conditions. To support national program planning, the Office of 
Education should help States to induce greater comparability into local 
Title I project evaluations and to pool available project evaluation 
resources into comparative evaluations of groups of projects attempting to 
deal with the same problem. These cooperative local evaluations should make 
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use of the same output measures, the same definitions of project contexts 
and populations served, and the same methodology to estimate the relative 
outputs of comparable projects. 

Local Educational Agencies should work together to establish coopera- 
tive mechanisms for more uniform, comparable, objective evaluation of the 
degree of effectiveness of alternative compensatory education strategies 
under varying circumstances. Groups of school districts should organize 
to collect comparable output data for projects with similar objectives 
serving similar groups of children. This would assist project rating and 
monitoring activities, greatly strengthen the Project Descriptor effort 
within the Belmont system, and could provide the basis for subsequent 
program strategy evaluation. 

One problem which is common to evaluation efforts at every level is 
the inadequacy of existing evaluation methodology for the Title I program. 

As a result, most evaluations of federally aided compensatory education 
programs have failed to meet even minimal requirements. The best of the 
Title I project evaluations done to date usually are little more than 
educational audits designed to meet annual deadlines set by the Office of 
Education and State Departments of Education. As increased emphasis is 
devoted to setting project objectives in measurable terms, and to encourag- 
ing evaluation against these objectives, the identification of research 
instruments currently available to assess the extent of progress toward 
Title I f s many objectives should be of great value to LEA’s, SEA’s, and 
the Office of Education in both their evaluation and technical assistance 
efforts . 

Traditional evaluations of Title I projects have relied primarily on 
tests of cognitive skills to measure the impact of the whole range of 
Title I supported activities. In part, this results from a lack of agree- 
ment on what kinds of evidence, other than test scores, would be acceptable 
as measures of educational impact; however, the preeminence of cognitive 
tests also results from the fact that evaluators lack the tools necessary 
to measure non-cognitive growth with equal reliability and confidence. It 
is clear that instruments must be developed and/or tested which focus on 
affective measures of student performance, as well as measures of institu- 
tional change associated with the introduction of compensatory education 
programs. There is a need for the Office of Education and, in turn, the 
States and local districts to agree upon and to define more clearly 
objectives for the Title I program. The Office of Education should make 
clear the kinds of evidence it would require in order to know whether 
Title I is working well. This will necessitate a careful sorting out of 
numerous objectives articulated in the Title I legislation and in project 
proposals from local school districts. The Office of Education should 
support the development of more sophisticated research instrumentation and 
evaluation to measure progress toward Title I goals, and should play a 
major role in developing new measures for assessing the effects of the 
Title I program on students, teachers, and on the institutions of education. 
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Both educators and researchers should stop behaving as if there were 
only one kind of evaluation (that results in a program's continuation or 
phasing out) and should work to develop more appropriate strategies for 
ongoing evaluations--where the questions of continuing the program is not 
at issue, but where the evaluative function is to identify the strengths 
and weaknesses of the elements in a program with a view towards improvement. 
Seen in this light, evaluation activities become the keys to the identifi- 
cation of specific areas where technical assistance can aid a program or 
project . 

Technical assistance has generally meant providing Federal advice and 
technical consultation to State and local "partners " The technical assis- 
tance process for compensatory education programs, however, should not be 
a rigid, one-way, vertical effort (from the Federal to the State and/or 
local levels), but a cooperative process to provide help and guidance to 
meet needs at all levels in the development and operation of compensatory 
education programs. The Federal government need not always be the provider 
of technical assistance to the States and through them to school districts. 
Although the Federal government will generally be the logical source for 
assistance in the interpretation of legal requirements, State and Local 
Educational Agencies should often be sources of assistance in program 
planning and development, assistance in the improvement of project opera- 
tions, and assistance in the design and use of evaluation. Thus, every 
level of educational administration--Federal , State, and local--may both 
offer and receive assistance in improving program operations. 

Since the enactment of Title I, the type of technical assistance 
offered has been primarily passive in nature, flowing from the Office of 
Education down through the States to school districts in response to 
requests for clarification about how Title I funds may be used. Too little 
attention has been devoted to providing assistance in improving project 
planning, operations, or evaluation. At all levels of Title I administra- 
tion, more emphasis should be placed on the improvement of the content, the 
management, the operations of compensatory projects and on the strengthening 
of evaluation activities. 

Federal technical assistance efforts have centered primarily on the 
interpretation of guidelines and on responding to questions concerning 
project applications. Federal technical assistance in the Title I program 
should be diversified: the Office of Education has an important role to 

play in improving program planning and development through the dissemination 
of tested methods or promising techniques for working with disadvantaged 
children; OE can be a resource for the improvement of management techniques 
useful to both State and local personnel; and OE should offer active tech- 
nical assistance to bolster the State role in helping school districts to 
collect and use evaluative data, to design and implement evaluation studies, 
and to assess the effects of evaluation on the direction of Title I projects. 

At the present time, the Office of Education has no mechanism for eval- 
uating the success of existing technical assistance efforts. Little attempt 
has been made by the Title I program to discover what types of technical 
assistance mechanisms exist in other Federal or State compensatory programs; 
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hence, little or no coordination exists among those programs. Moreover, 
the Office of Education lacks adequate procedures f^r determining needs 
for technical assistance. 

To improve this situation, the Office of Education, in cooperation 
with several State Educational Agencies, should develop and evaluate 
several models for the delivery of technical assistance within a State. 
Although each model would include a means of determining what types of 
assistance are needed, the services provided and the techniques used would 
vary. In addition, the Division of Compensatory Education should assess 
annually State and local technical assistance needs, should review the 
general programmatic weaknesses which have been uncovered, and should 
reappraise its own staff capability to meet the needs identified. 

The Office of Education should develop, as soon as possible, a means 
of assessing the impact of the technical assistance it provides. Criteria 
for determining which types of technical assistance work best should be 
developed as a joint Federal-State-local effort. In addition, the Office 
of Education should require that States include in their annual report an 
assessment of their technical assistance efforts, with appropriate plans 
for evaluating the following year's technical assistance efforts. 

Like Federal efforts, State assistance to local Title I projects tend 
to be informal and inadequate. State efforts are constrained by lack of 
resources, lack of staff skills, and absence of clearly developed objectives 
and priorities for technical assistance and a lack of methods for evaluating 
the assistance provided. State Educational Agencies should overcome their 
n one-dimensional M view of technical assistance, i.e., that technical assis- 
tance is always delivered vertically from the State to local school 
authorities. Instead, States should consider providing and receiving 
technical assistance horizontally. Not only should they try to initiate 
cooperative arrangements among school districts, but they also should try 
to develop cooperative arrangements with other States in order to share 
expertise among State personnel. Such approaches would not only develop 
staff competencies in the agencies involved but also enable the States to 
provide technical assistance more efficiently and systematically to groups 
of clients. 

No systematic attempt has yet been made to assess the impact of tech- 
nical assistance provided to the State Departments of Education or the 
technical assistance provided by the States to school districts. State 
Department staffs could make a substantial contribution by identifying 
their greatest needs and providing data on the extent to which these needs 
are being met by various types of technical assistance provided by the 
Office of Education. The State authorities can also help by identifying 
mechanisms that they have used to provide technical assistance to school 
districts (and any other clients) and by evaluating the efficacy of their 
technical assistance efforts. 

The large school districts represent an important potential resource 
for providing technical assistance. They often possess resources and 
technical skills which are at least equal to those of Federal and State 
agencies. Insufficient attention has been given to the capabilities of 
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school systems to provide other school systems. State Departments, and the 
Office of Education with technical assistance in some important areas. It 
would be worthwhile to develop and demonstrate new mechanisms which would 
enable local districts to cooperate with the Federal and State levels by 
sharing the expertise they have developed in providing technical assistance 
to their own local projects. Some larger school districts may possess the 
capabilities to deliver technical assistance directly to other districts. 

And local school officials appear well equipped and willing to provide the 
Office of Education with basic information necessary to develop and expand 
technical assistance mechanisms. 

The Division of Compensatory Education should help local districts to 
develop cooperative arrangements to increase technical assistance expertise-- 
both their own and that of the State. Funds available under the Education 
Professions Development Act might be earmarked specifically for in-service 
training in the development and provision of various kinds of technical 
assistance services. The Office of Education should support pilot grants 
to local districts willing to demonstrate cooperative technical assistance 
arrangements with other school districts. 

The Elementary and Secondary Education Act of 1965 was the first major 
piece of education legislation to require annual evaluation of program 
effectiveness. Federal, State, and Local Educational Agencies are all 
involved in assessing Title I project effectiveness and in attempts to 
improve program operations. This report emphasizes the need for the Office 
of Education to strengthen State Title I evaluation and technical assistance 
in order to improve the education of disadvantaged children--and the need 
for the Office of Education to develop and test evaluation and technical 
assistance models that are appropriate to different educational settings. 
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II. FEDERAL TITLE I EVALUATION: 
PRIORITIES , PRESENT STATUS, AND NEXT STEPS 



This chapter examines the major Federal, State, and local decisions 
related to education of disadvantaged children, the most important types 
of evaluative data that could assist these decisions, relationships 
between decision timing and the times required for the evaluation, and the 
methodological feasibility of each of the major types of Title I evalua- 
tion. The chapter includes a discussion of the organization of Federal 
efforts to evaluate Title I, an assessment of the Belmont system, recom- 
mendations on how the Office of Education can improve existing Title I 
evaluation efforts, and recommendations on useful new directions for 
Title I evaluation. 

A. Introduction 



Evaluation is a complex and often expensive undertaking which attempts 
to provide objective information about the operations, costs, and effects 
of national programs, and of local projects. Evaluations differ from one 
another in size, scope, purpose, design, level of complexity, and costs. 

But all evaluations share one common element: each seeks to improve the 

current operations of the program being evaluated and to have an impact on 
plans for future programs. 

For the purposes of this report, we have identified eight common types 
of evaluation activities 1/: (1) program impact evaluation; (2) program 

strategy evaluation (including evaluation of sets of field experiments and 
sets of experimental demonstration projects designed to test the relative 
effectiveness of alternative strategies and techniques within a program); 

(3) project rating; (4) local project evaluation; (5) monitoring; 

(6) program reporting; (7) needs assessment; and (8) cost analysis. The 
appropriateness of each type of evaluation for education programs will 
depend on the type of decision to be affected, the availability of data 
for conducting the evaluation, and the resources available (manpower, money, 
and methodology) . 2 J 

Evaluation is a mandated activity in the Title I program, and every 
level of government is involved. Yet, evaluation of the Title I program 
is extraordinarily complex. At the local project level, where all evalua- 
tions must start, Title I funds can be used to finance a broad range of 



\/ For a more extensive discussion of each of these activities, see 
Appendix B. 

2/ See Appendix C for a discussion of the types of data needed for deci- 
sions regarding the education of disadvantaged children. 
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activities from preschool through high school. Individual projects may 
differ in almost every dimension. Levels of funding vary from a few 
dollars to several hundred dollars per child--but the Title I expenditures 
are ordinarily relatively meager compared to regular expenditures from 
State and local funds. These are but a few of the problems which confront 
those who attempt to evaluate Title I projects. Regardless of these 
difficulties, evaluation activities continue. 

This chapter is concerned with what kinds of evaluation are appro- 
priate for education programs for disadvantaged children and for the 
Title I program in particular. It attempts to establish a framework for 
evaluating a program which operates in all 50 States and in more than 
16,000 local school districts, a program whose planning and administration 
involves the Executive and Legislative branches of the Federal government, 
State legislatures and Departments of Education, local school officials 
and parents, as well as interested citizens and the ultimate beneficiaries-- 
the students. The improvement of management and operation of evaluation 
activities at the Federal level is the primary concern of this chapter, 
although there are some discussions of how State and local evaluation 
efforts can be strengthened. 

B. Types of Evaluation which can Assist Decisions 

Relating to the Education of Disadvantaged Children 

Evaluations of the Title I program--and of all education programs for 
the disadvantaged--are designed to influence a series of decisions concern- 
ing the amount and type of services to be provided. Evaluative information 
can be useful for the following decisions: 

1. The amounts of Federal, State, and local appropriations/ 
expenditures for elementary and secondary education; 

2. The allocation of general education funds among school 
districts and among schools within a district; 

3. The allocation of funds for the education of disadvantaged 
children among school districts and among schools within 

a district; 

4. The selection of types of local projects for the education 
of disadvantaged children; 

5. The selection of staff for the education of disadvantaged 
children; and 

6. Decisions concerning the day-to-day operations of educa- 
tion programs for disadvantaged children. 

The key decision-makers involved in these decisions vary, depending 
on the level at which the decision is made. Similarly, the types of 
evaluation which could assist these decisions vary. Table II-l lists the 
major decisions related to the amounts and types of services provided for 
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a/ Types of evaluation are ranked from most important _/(l^/ to least important for each of the decisions examined (see Appendix B for definitions 
of types of evaluation). These rankings are not affected by considerations of present availability of data. 
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disadvantaged children teachers’ associa- 
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the education of disadvantaged children, identifies the major decision- 
makers involved in each, and summarizes the relative importance of each 
type of evaluation. 3/ 

An examination of the information presented in Table II-l leads to 
several observations about the types of evaluation activities most appro- 
priate to the Title I program. Regardless of the decision or the 
decision-makers involved, program reporting, needs assessment, and 
monitoring (assessment of the managerial and operational efficiency of 
programs and projects) emerge as the most needed Title I evaluation 
activities. The reason for this is apparent: before the most basic 

decision can be made (or before the most sophisticated evaluation can 
take place), information must be available about the populations served, 
services provided, the costs of providing services, and the relationships 
among measures of educational need and contextual variables. Similarly, 
decision-makers at every level need to have some assessment of the 
operational efficiency of projects and programs. Project evaluation/ 
project rating and program strategy evaluation come next in importance 
for decisions concerning the allocation of funds and services for disad- 
vantaged children. These types of evaluation should focus on relative 
project or program success--inf ormation which is critical to decisions 
about the most effective allocation of services and scarce resources. 
Program impact evaluation, which is concerned with the effectiveness of 
an entire program, is important only for global decisions on total levels 
of funding. Program impact evaluation in the Title I program itself is 
of relatively little use either to local or to State education decision- 
makers . 

C. Priorities for Federal Title I Evaluation Efforts 



The foregoing discussion exclusively focused on the appropriateness 
of the different types of evaluation to decisions and decision-makers at 
all educational levels, rather than on the types of evaluation which the 
Federal government should pursue in the Title I program. The remainder 
of this chapter will deal primarily with the Federal government's role in 
evaluating education programs for the disadvantaged and the Title I program 
in particular. 

Despite increases in the past few years in the amount of money avail- 
able to evaluate Federal education programs, resources are still in short 
supply. Much less than 1 % of the Office of Education's budget is used in 
support of its evaluation activities. And even if the available funds 
doubled or tripled, evaluation efforts would still be hampered by shortages 
of personnel, by limitations in existing methodology, and by the availability 



3 / 



See Appendix C for an expanded discussion of the types of evaluation 
which can assist decisions relating to the education of disadvantaged 
children . 
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and usability of data on many important aspects of education programs. 

Given all of these limiting conditions, it is important to determine how 
best to use available Federal evaluation funds to generate useful infor- 
mation for decision-makers. Seen in this light, the eight types of 
evaluation identified in this report become means by which to influence 
decisions about the Title I program. The results of each type of evalua- 
tion can affect several different decisions. The appropriateness of each 
type of evaluation depends on. several factors: (a) how often the policy 

decisions to be affected are made; (b) how long it takes to carry out the 
particular type of evaluation; and (c) whether the evaluation is 
methodologically feasible. Table II-2 takes each type of evaluation and 
summarizes (a) the decisions which could be affected by that evaluation; 

(b) the relationships between decision timing and the time required for 
evaluation; (c) the present availability and methodological feasibility 
of obtaining the necessary evaluative data; and (d) the tentative priority 
for that type of evaluation. 

Table II-2 points up a fact about evaluation that is often overlooked: 
the time required to carry out an evaluation and the time of the decision- 
making cycle often do not coincide. In discussing evaluation, very often 
one really means the results of an evaluation and not the whole process 
of evaluation, from design through data collection and analysis. Evalua- 
tion is more than just a written report--it is a process--of ten a time- 
consuming process* Twelve months is probably the minimum time necessary 
for the Office of Education to design and carry out an evaluation, which 
involves the collection of data on an entire school year. At this stage 
in Federal evaluation efforts, 12 months is an ideal rarely achieved. 
Decisions, on the other hand, are often made annually; and decision-makers 
continue to pose questions which cannot be answered in 12 months. This 
inability of evaluation to respond rapidly to the needs of decision-makers 
makes more difficult attempts to build support for the process of evaluation. 

Similarly, many evaluations cannot be done in time to fit annual 
cycles (are harmed if one tries to tailor them to crash deadlines) , and 
need not be done annually to be useful. In assigning priorities to 
evaluation, officials should take into account what decisions are to be 
affected, and how much time each type of evaluation requires. 



When all of the factors cited in Table II-2 are taken into account, 
the following priorities emerge for Federal Title I evaluation efforts: 



High 


Relatively High 


Moderate 
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Low 
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Project Rating 3. Allocation of funds for Annual cycle Relatively short: Feasible given availability Project rating should have a 

education of disadvantaged one month to one of suitable (short-term) relatively high priority for 

children among school year (once the output measures and measures Title I. It is a relatively 
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1. Program impact evaluation . At the present time, Title I program 

impact evaluation should have low Title I evaluation priority among Federal 
evaluation efforts for two reasons: (a) it is difficult or impossible to 

separate the effects of low levels of Title I expenditures from the effects 
of other expenditures on education of disadvantaged children; (b) Title I 
projects are only rarely designed with suitable control or comparison 
groups. Although appropriation decisions which might be affected by program 
impact evaluation are made annually, this type of evaluation generally takes 
more than 12 months to execute. 

2. Program strategy evaluation * Program strategy evaluation is a 
high cost, low payoff investment within Title I and should therefore have 
a relatively low Title I evaluation priority. Program strategy evaluation 
should have a much higher priority within those Federal, State, or local 
compensatory education programs (such as the Follow Through program) in 
which systematic attempts are made to test the relative effectiveness of 
alternative approaches to education of disadvantaged children. Evaluation 
of the relative effectiveness of different compensatory program strategies 
should be longitudinal in design; should take into account all major school 
programs, not just compensatory programs; should gather data on the social 
class of individual students, classrooms, and schools; must use systematic 
and consistent testing; should describe programs in sufficient detail to 
allow identification of variations in inputs among grades and among schools; 
and should develop techniques for expenditure accounting by school and, if 
possible, by grade within schools. 

The Title I program was not designed as a vehicle for systematically 
testing alternative strategies for compensatory education. Although a 
researcher could no doubt define several program strategies and find each 
strategy in operation in several school systems, the Office of Education 
has no direct control over their operation. Program strategy evaluation 
would therefore be difficult and would probably not be definitive in the 
Title I program, where inputs and organization of inputs are difficult to 
define and measure and where effects of program components will be inter- 
mingled, in unknown ways, with effects of other forces. 

3. Project evaluation and project rating . Local project evaluation 
is an integral part of the Title I program, but its utility for decision- 
making is generally diminished because the results of these evaluations 
are neither comparable nor timely. Although the precise amount spent on 
local project evaluation is not known, an estimate of $10 million per year 
is not unreasonable. Most local evaluations are undertaken to fulfill the 
mandated requirements of ESEA and are often not completed in time to be of 
use in local decision-making. For local decision-making, evaluations of 
individual local projects should focus on relatively inexpensive short-term 
comparisons of project results with previously stated performance objectives 
or with baseline conditions. To support national program planning, the 
Office of Education should help States to induce greater comparability into 
local Title I project evaluations and to pool available project evaluation 
resources into comparative evaluations of groups of projects attempting to 
deal with the same problem. These cooperative local evaluations should 
make use of the same output measures, the same definitions of project con- 
texts and populations served, and the same methodology to estimate the 
relative outputs of comparable projects. 
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Introduction of greater comparability among local project evaluations 
would lead to the possibility of project rating. Project rating has not 
been done in Title I to date but is feasible for groups of Title I projects 
that have similar objectives, operate in similar environments, and serve 
similar populations- -if the projects can be induced to use some comparable 
output measures (preferably short-term output measures) . 



4. Monitoring . Systematic monitoring of Title I projects can assist 
the whole range of decisions from the global to the specific. Present 
Title I monitoring efforts are quite deficient. Some typical problems 
with current monitoring activities are the following: (a) assessments are 

too subjective, relying on impressions gathered through unstructured inter- 
views; (b) assessments are not reliable enough to allow comparisons among 
groups of projects; (c) monitoring personnel receive little or no training; 
(d) there is no systematic follow-up to determine what changes are made in 
project operations; (e) there is no system for funneling Title I program 
monitoring information from the States to the Office of Education. 



The Title I program needs systems for State Educational Agency 
classification and monitoring of Title I projects, perhaps combining 
assessments of degree of project compliance with Federal and State regu- 
lations and guidelines, assessment of project quality , and financial 
audits. Division of Compensatory Education monitoring of SEA's should 
also have a high priority for Title I. 



Increasing the scope of audits conducted by audit offices to include 
checking compliance with Federal and State regulations and guidelines, 
both financial and non-f inancial , seems feasible and desirable. In addi- 
tion, the Office of Education should experiment with increasing the scope 
of the annual financial audits contracted for by the districts, to include 
checking compliance with Federal and State regulations and guidelines. 

5. Program reporting/ needs assessment . These activities ought to 
have high priority for Federal Title I evaluation efforts, since not only 
will they generate useful descriptive information about the program, but 
they will also provide baseline information needed for the design of other 
types of evaluation. 



The time required to gather program reporting and needs assessment 
data varies from a few months to a year or more. There are important 
practical problems in ensuring that these data are timely, reliable and 
relevant--and not so voluminous as to make the system for data collection 
more costly than it is worth. 



The collection of program reporting and needs assessment data should 
have high priority among Federal evaluation activities, but should not be 
pursued to the exclusion of other evaluation activities. It may be suffi- 
cient to collect some program reporting/needs assessment data on a two-year 
cycle, thereby freeing limited evaluation funds to be used on other types 
of evaluation. 
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6. Cost Analysis . Comparative cost analyses can be a valuable tool 
in providing information on comparability of local expenditures among 
schools within a district. Accurate information on relative costs of 
various types of projects can also assist decisions on project selection, 
even in the absence of data on relative project effectiveness. 

There are methodological problems involved in gathering comparable 
information on school-by-school expenditures. The Office of Education 
should develop cost analysis/cost estimating systems which could be used 
to determine the extent to which local services are being made comparable 
among Title I and non-Title I schools. 

D. The Organization of Federal Evaluation Efforts 

Many different units within the Office of Education and several 
outside OE have a part in Federal evaluation activities. From time to 
time, the Office of Management and Budget (formerly the Bureau of the 
Budget) plays an active role in defining issues to be investigated in 
subsequent evaluations, but to date, this agency has not conducted its 
own evaluations independently of OE . The Elementary and Secondary 
Education Act created the National Advisory Council on the Education of 
Disadvantaged Children (NAC) and gave it some responsibility for evaluating 
the Title I program. During the first years of its existence, the NAC 
sent observers out into school districts across the country to record and 
report on the progress and problems of Title I. In recent months, the NAC 
has been inactive; its last report was issued in January 1969. Congressional 
committees have also been active from time to time in gathering information 
about the Title I program. 

While important, all of these activities are clearly ancillary to the 
evaluation activities carried on directly by OE or its parent organization, 
the Department of Health, Education, and Welfare. During the five years 
since the enactment of Title I, evaluation responsibilities within OE/HEW 
have shifted almost rhythmically, altering the role played by each admin- 
istrative unit. Federal evaluation activities fall into two rather distinct 
phases: Phase I covers the first two years of the program* s life and repre- 

sents the time of ascendancy of the Division of Compensatory Education (DCE) 
in the formulation and execution of Title I evaluation. Phase II begins 
with the decision, in late 1967, to conduct a national survey in order to 
satisfy the mandated evaluation requirement and coincides with the central- 
ization of Title I evaluation responsibilities in the Bureau of Elementary 
and Secondary Education (BESE) . Not only did the organizational responsi- 
bilities for Title I change, FY 1968 marks a change in the substantive issues 
of evaluation. Emphasis shifted from the evaluation of Title I-funded 
projects to the evaluation of education programs for the disadvantaged, 
regardless of source of funding.. 

Today, the evaluation activities for the Title I program are primarily 
the responsibility of BESE, and are subsumed under the umbrella of the 
"Belmont system." kj Although Belmont has accounted for a large share of 



kj The Belmont system is discussed in some detail in the next section. 
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the money appropriated specifically for evaluation, several administrative 
units in addition to BESE have some involvement in Title I evaluation 
activities as we have defined them in this report. A brief description of 
the Title I evaluation activities of each of those units follows. 

1. Division of Compensatory Education (DCE) . Although DCE has no 
official evaluation responsibility, it continues to influence the direction 
of the Title I program through the development of guidelines and the issuance 
of instructions concerning all aspects of project administration. In recent 
months, DCE has begun a series of Program Reviews or monitoring visits to 
State Departments of Education. 

2. Bureau of Elementary and Secondary Education (BESE) . The Office 
of Program Planning and Evaluation in BESE has primary responsibility for 
the development and operation of the Belmont system for evaluation. 

Title I is one of several Federal education programs covered under the 
Belmont system. Prior to FY 1970, the Belmont system gathered primarily 
needs assessment /program reporting data. Present plans for the Belmont 
system call for the collection of other types of evaluative data. 
Coordination between BESE and DCE in the development and implementation of 
the Belmont system has been far from systematic. 

3. Office of Program Planning and Evaluation (OPPE) . This unit 
functions at the Commissioner^ level and has overall responsibility for 
all OE evaluation activities. OPPE has, in the past, studied particular 
aspects of compensatory education, such as the AIR study of "successful" 
compensatory projects and the in-house OPPE estimates of achievement 
success and failure cited in the 1970 Presidential message on education. 

OPPE must approve the expenditure of all funds earmarked for evaluation. 

4. National Center for Educational Statistics (NCES) . The NCES, by 
virtue of its forms clearance responsibility, serves as a control in the 
development of evaluation instruments. During the early years of Title I, 
the NCES was responsible for several attempts to develop cost/benefit models 
for the program. NCES staff are sometimes called upon to draw samples for 
evaluation surveys, and some of NCES 1 own survey activities yield informa- 
tion used in Title I evaluation. 

5. Division of State Agency Cooperation (DSAC) . This Division, which 
is a part of BESE, monitors the operations of State Departments of Education 
through its State Management Reviews (SMR's). Each SMR is a cooperative 
Federal-State review of State Agency operations along functional lines. 

Title I is included in the review, along with other Federal programs. 

6. Assistant Secretary for Planning and Evaluation (ASPE) . This unit, 
in the Office of the Secretary of HEW, oversees all evaluation activities 

in the Department. It has control over a portion of the available education 
evaluation funds, which are used in support of evaluation and monitoring 
studies of its own design and choosing. 
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?• HEW Audit Agency 5/. Although they have some serious limitations, 
audits conducted by HEW audit staff have been a very important source of 
information on what is going on in Title I projects. HEW auditors visit 
some SEA's and a sample of LEA's during the course of a year. Their reviews 
include a check on SEA and LEA compliance with Title I financial and non- 
financial guidelines . 

E. Present Federal Evaluation Efforts: The Belmont System 

In FY 1970, the budget for the Office of Education included a total 
of $14.5 million earmarked for support of planning and evaluation 
activities. These funds were spread over all the activities of 0E , but 
the largest portion, $8.8 million was appropriated for elementary and 
secondary education activities. Of this money, $5 million was to be 
distributed to the States and the remaining $3.8 million was to be used 
for 0E evaluation studies. 6/ (In FY 1969, 0E spent only $3 to $4 million 
on evaluation; of this total, approximately $600,000 was spent on evalua- 
tion contracts relating to Title I.) 

In addition to funds specifically earmarked for evaluation, evaluation 
activities can be supported by other appropriations. For example, the 
Office of Management Information (0MI) is responsible for data processing 
in 0E and sometimes provides funds for the processing of evaluation survey 
data. Monitoring activities of the HEW Audit Agency and the Division of 
State Agency Cooperation and the Division of Compensatory Education are paid 
for out of regular "salaries and expenses." 

Funds for evaluation are administered by the Office of the Deputy 
Assistant Secretary for Planning, Research, and Evaluation (DASPRE) in 0E. 

By mutual agreement, the Assistant Secretary for Planning and Evaluation 
(HEW) administers 25% of the evaluation funds. For the remainder, the 
Bureaus within 0E develop evaluation plans describing what funds are needed, 
what activities will be supported, and what evaluation questions will be 
answered. The evaluation plans are reviewed and approved by the Office of 
Program Planning and Evaluation, part of DASPRE. 

In FY 1970, contracts for the Belmont system amounted to approximately 
$2.7 million, of which $1.8 million came from funds available for 0E eval- 
uation. 7/ The Belmont system is clearly the major Federal evaluation 
activity relating to elementary and secondary education. (Local project 
evaluations account for the largest evaluation expenditure related to 
Title I, however.) Belmont accounts for nearly all the staff time of the 



5/ See Appendix F for a more detailed discussion of the possibility of 
monitoring Title 1 through audits. 

6/ This $3.8 million could be augmented by part of the $2.8 million 
appropriated for 0E evaluation in general. 

7/ BESE/PPE estimate. 
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BESE planning and evaluation office. Except for monitoring activities 
carried on throughout OE/HEW and for a few contracts awarded by OPPE or 
ASPE, Belmont accounts for all the OE evaluation related to Title I (see 
Table II-3) . It is for these reasons that we now take a closer look at 
the Belmont system. 

The Belmont system is a cooperative OE-State evaluation effort, begun 
in late 1968. 8/ To date, 27 States participate in the operation of the 
Belmont system. The programs to be evaluated under the Belmont system 
include: ESEA Titles I, II, III, V (Flow-Through funds only), VII, and 

VIII; NDEA Titles III and V-A; Follow Through; and programs under the 
Civil Rights Act Title IV. Programs of the Bureaus of the Handicapped and 
of Vocational and Technical Education are to be added. The goals of the 
Belmont system are as follows: 9/ 

(1) to provide the capability for accurate assessment of 
school and pupil educational achievement, 

(2) to evaluate the differential effectiveness of the 
various federally funded programs, 

(3) to evaluate the process of funds distribution through 
the delivery system (the overall system concerned with 
the transmittal of funds from funding sources through 
to substantive programs) , 

(4) to investigate the degree to which various needs of 
the students, local agencies, and schools are being 
met through current Federal contributions in conjunc- 
tion with local and State financing. 

These goals have been translated into six objectives for the purpose of 
directing instrument development and implementation: 10 / 

"(1) Describe individual school characteristics and pupil 
characteristics at the school level; 

"(2) Describe the projects, activities, or treatments in 
which pupils are participating at the school level; 

"(3) Relate school and pupil characteristics to specific 
projects, activities, and treatments; 



8/ The precise title is the Joint Federal/State Task Force on Evaluation. 
The name "Belmont" derives from the location of the first planning and 
development sessions . 

9/ Scientific Educational Systems, Inc., Joint Federal/State Task Force 
on Evaluation, Comprehensive Evaluation System^ Current Status and 
Development Requirements , Draft, January 1970, p. 17. 

10/ Ibid. , p. 18. 
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TABLE II-3 



Estimates of Title I-Related Evaluation Expenditures 
(Fiscal Year 1970) 



Total 



Total 



$14,876+ ? 



Program Strategy Evaluation 

Project Rating ) 

) 

Local Project Evaluation ) 
Monitoring 

Program Impact Evaluation ) 

) 

Program Reporting/Needs ) 
Assessment ) 

Comparative Cost Analyses 

Other 



364+ ? 
10,321+ ? 
205+ ? 



3,258+ ? 
? 



728+ ? 



Federal State Local 

(thousands of dollars) 



$2,876+ ? 
364 d 

321 y 

205 y 
1,258 - 1 
? 

728 y 



? $12,000+ ? 

? ? 



? 10,000+ ? y 

? 

? 2 , 000 + ? 



a/ Longitudinal Study of Demonstration Education Programs (OPPE) , Further 

Examination of Exemplary Elementary and Secondary Reading and Early Child- 
hood Programs (OPPE) . 

b/ Studies of Effectiveness of Elementary and Secondary Programs (Belmont 
Project Descriptor effort). 

c/ Development and Installation of SEA Management Appraisal System (Belmont) . 
d/ Studies of Impact of Elementary and Secondary Programs (Belmont contract 
awards for BESE/PPE projects in fiscal year 1970). 
e/ Local expenditures are estimated to be at least 1% of local project 
expenditures. 

f/ Costs of completing the Belmont teacher and pupil surveys are said to 
exceed 8 hours /teacher (x 22,000 teachers); costs of completing the 
Consolidated Program Information Report will be very substantial, 
g/ Contracts for development of Belmont analysis plans and user guides, 

Belmont contracts for development of common output measures and an "anchor 
test," etc. 





28 



"(4) Describe overall program progress; 

M (5) Identify successful projects, activities, or 
treatments; and 

"(6) Assess needs in terms of (a) flow of service, 

(b) populations being served, (c) projects, 
activities, or treatments not being provided, 
and (d) pupil changes not occurring." 

The development of the Belmont system has been guided by two factors: 
cooperation with the States and consolidation of existing evaluation and 
statistical reporting requirements. The data collection needed to support 
this management and evaluation system has been developed with emphasis on 
removing redundant reporting requirements and minimizing requirements for 
new information. 

Table II-4 lists the instruments which make up the Belmont system, their 
purposes, the data to be collected, and the present status of each. In addi- 
tion to the instruments described in Table II-4, Belmont is pursuing three 
alternative approaches to the development of output measures for compensa- 
tory education programs: (1) development of common status measures, short 
tests of reading achievement and knowledge of job opportunities/requirements, 
for fourth and eleventh grade levels; (2) development of a new criterion 
reference instrument for reading; and (3) development of a common reading 
achievement metric ("anchor test") for fourth, fifth, and sixth grade levels, 
based on reading subtests of the SAT, MAT, STEP, SRA, and ITBS achievement 
batteries. The common status measures have been field-tested in Colorado 
this spring. A final report on this project is due September 30, 1970. 

Development of a common reading achievement metric depends on 
(a) restandardization and development of common norms on a battery of five 
reading achievement subtests, (b) development of tables of score corre- 
spondence for the five reading subtests using the anchor test approach, 
and (c) the administration of the total testing program in the same time 
period to achieve comparability. 11 / The work is expected to take at least 
15 months . 

Table II-5 relates the principal Belmont instruments to the types of 
evaluation discussed in Section A of this chapter. Inspection of Table II-5 
reveals that the Consolidated Program Information Report and the Pupil- 
Centered Instrument will be useful for Title I program reporting, needs 
assessment, and possibly for program impact evaluation; the Project 
Descriptor Workbook and the Guide for Authors will be useful for project 
rating and project evaluation; but that Belmont has less applicability to 
program strategy evaluation, monitoring, and cost analysis. 



11/ U. S. Office of Education draft RFP, "Development of a Common Reading 
Achievement Metric Based on Reading Subtests of the SAT, MAT, STEP, 
SRA, and ITBS Achievement Batteries," April 9, 1970. 
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*There is some doubt that the detailed breakdown of pupil, staff, and expenditure data are reliabily available 
for LEA's at this time. 
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The components of the Belmont system are discussed in more detail in 
Appendix D. Based on the review of materials related to Belmont, the 
following observations can be made: 

1. Belmont is a promising effort at cooperative Federal-State data 
collection and evaluation, which is potentially very useful for Title I 
evaluation. The States are pleased with their partnership role in Belmont. 
Whenever possible, therefore, new cooperative Title I evaluation efforts 
should be launched under the Belmont umbrella. 

2. Belmont has developed a number of data collection instruments 
without having a clear data analysis plan and plan for use of the data. 12 / 
Belmont looks like a M big system,'* in danger of being designed relatively 
independently of potential users of the system, not furnishing enough 
short-term help to motivate potential users to invest the staff time 
required to make the system relevant and, therefore, in danger of being a 
waste of money. Belmont was started in the right direction, however. With 
sufficient staff for Belmont and liaison staff from the operating program 
and State and large city agencies, Belmont could still be a model of 
cooperative development of data collection and evaluation systems of use 

to large numbers of Federal, State, and local personnel. 

3. Office of Education evaluation efforts, and Belmont evaluation 
efforts in particular, are suffering from lack of full-time professional 
staff members. The Belmont manager estimates that the following profes- 



sional staff members are involved in 


Belmont : 






Total 


Full-time Equivalent 


Bureau Program Planning and 
Evaluation (BESE, BAVTE , BEH) 


25 


12 


Other 0E/0S 


25 


. 7 


SEA 


85 


22 



While these numbers represent a large investment of OE staff in the 
Belmont project, more staff are need to ensure that contracts are properly 
monitored. 13 / 



12 / "... At present no one in Belmont has the time to explore the numerous 

dimensions or possibilities for use of the data being collected .... 
Belmont provides information for decision makers at all levels of 
federalism. But these potential users can best become actual users 
if they understand the nature and utility of the Belmont data. This 
report has stressed that the utility of Belmont data is not obvious 
to many LEA's who will bear the brunt of filling out the surveys" 
(Michael Kirst, Proposed Management System for Belmont , June 12, 1970 
(OEC -0-70-2920) . 

13 / Cf. Michael Kirst, op . cit . , "... Belmont has relied heavily on outside 
groups for doing interpretations and even drafting reports to Congress 
.... It is unlikely that private contractors will know the audience 
or policy or program issues most salient to Federal, State, and local 
decision makers." 
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4. The Belmont system furnishes data which could be used to estimate 
the extent to which Title I services are going to those with the greatest 
needs; the Consolidated Program Information Report makes it possible to 
trace funds to school district level; the Pupil-Centered Instrument makes 
it possible to estimate levels of services added to individual schools. 

5. The recent inclusion of non-Title I schools in the sample used in 

the Pupil-Centered Instrument will furnish data that could be used in 
attempts at program impact evaluation, though the lack of appropriate com- 
parison groups, lack of follow-up data, and the low level of Title I inputs 
will handicap attempts at program impact evaluation. [An early effort is 
indicated to extend the Pupil-Centered Instrument beyond grades 2, 4, and 
6: upward to a Secondary School Survey (now scheduled for 1972) and down- 

ward to preschool and kindergarten.] 

6. The Project Descriptor, one of the most promising of the Belmont 
efforts, will result in collection of comparable evaluative data on large 
numbers of local projects. It will be possible to automatically sort 
projects into groups of comparable projects with similar objectives, 
operating in similar contexts, and serving similar types of children. The 
Project Descriptor Instrument could be improved if comparable output 
measures were used at least for potentially successful projects. Descrip- 
tions and results of such projects could be used for nationwide or statewide 
dissemination. 

The following are recommendations concerned with ways to strengthen 
the Belmont system and to make it more useful and responsive to the needs 
of Title I program administrators: 

1. The Division of Compensatory Education should allocate staff time 
to work closely with the Belmont system in order to make it more useful for 
Title I program management. In particular, DCE should invest staff time in 
helping develop plans for analysis of Belmont data and in helping develop 
and modify the Consolidated Program Information Report, the Pupil-Centered 
Instrument (for elementary and secondary levels), the Project Descriptor 
Workbook, and the Common Status Measures. 

2. The Division of Compensatory Education should attempt to make use 
of data currently available through Belmont and should work to shape the 
development of the Belmont system to better meet the Title I needs related 
to reporting, needs assessment, and program impact evaluation. In view of 
Belmont capabilities for producing Title I program reporting and needs 
assessment data, and in view of the low priority Title I program impact 
evaluation should have at the present time, the Office of Education should 
not mount additional, independent Title I evaluation efforts of these three 
types . 



3. The Belmont system should proceed with development of criterion 
measures (measures of minimum performance levels) in each of the high- 
priority Title I areas, not simply reading but also mathematics, early 
childhood education, etc., and make the resulting instruments available to 
States and local school districts. 
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4. The Office of Education should examine ways to reduce the serious 
reporting burdens of the Consolidated Program Information Report and the 
Pupil-Centered Instrument, e.g., requiring completion of these instruments 
on only a biennial basis or making more use of centralized data processing 
in place of present manual computations. In particular, Belmont should 
examine the possible central use of computers to perform the cost estimating 
procedures now to be done manually at the local level to obtain the three- 
way 8 x 50 x 17-item breakdown of expenditures requested in the Consolidated 
Program Information Report. 

5. Belmont should move the Project Descriptor system toward a project 
rating system by encouraging Local Educational Agencies to supply comparable 
output data, at least on projects which are nominated by SEA’s or LEA’s as 
outstanding and possibly worthy of replication in other communities within 
the State or in other States. 

The Division of Compensatory Education should attempt to have developed, 
as part of the Project Descriptor effort, "sugges ted" output measures for 
some or all of the types of projects of highest priority within Title I. 

These ’’suggested" output measures should then be tested, on a voluntary 
basis, in projects nominated by States or by Local Educational Agencies. 
Addition of comparable output data would greatly strengthen the Project 
Descriptor system. The Division of Compensatory Education should put staff 
time into helping develop the Project Descriptor system, to ensure that it 
is as useful as possible for Title I project rating and project evaluation. 

6. In addition, since it is difficult or impossible to separate the 
effects of Title I projects from the effects of other local efforts to 
educate the same children, the Project Descriptor effort should include 
some attempts to treat schools as projects (i.e., as the units of observa- 
tion), trying to estimate which schools are doing the best job among 
schools operating in similar environments, with the idea of coming back 
later for more process-oriented (program strategy) evaluation. 

F. Long-Term Strategy and Immediate Next Steps 

in Office of Education Title I Evaluation 

Based on the evaluation priorities recommended above, on present 
availability of evaluative data, and on present capabilities within the 
Office of Education, this section recommends a long-term strategy and 
immediate next steps in Title I evaluation for the Office of Education, 
the Bureau of Elementary and Secondary Education, the Division of 
Compensatory Education, and the Office of Program Planning and Evaluation. 

1. Long-Term Strategy — ^ 

At least $3 to $4 million per year is available for Office of 
Education Title I evaluation, but relatively little effort is going into 
project rating or into improving Title I monitoring. Title I monitoring 



14 / The reader may wish to refer to Table II-2, p. 17. 
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efforts are quite deficient at the present time--hence fine-sounding 
Title I regulations and guidelines have often been ignored in practice. 

Most State and local Title I evaluation activity is carried out to 
meet Federal requirements and is not completed in time to be useful 
locally. Each year, more than $10 million of Title I funds is spent 
locally on individual, unrelated Title I project evaluations, most of 
which are noncomparable and inconclusive because of the low level of 
Federal inputs and because of the absence of suitable comparison groups. 
Such local Title I evaluation studies are useful neither for local nor 
for State or Federal decision-making. And the Office of Education is 
failing to make the most effective use of millions of evaluation dollars 
available at Federal level. 

Congressionally-mandated evaluations of the national impact of the 
Title I program, of alternative Title I program components, and of 
individual local Title I projects are all very difficult methodologically, 
because of the low level of Federal inputs relative to local inputs and 
because of the absence of suitable comparison groups. The Office of 
Education should give much higher priority to Federal and State monitoring 
of Title I operations and comparative rating of the relative outputs of 
Title I projects with similar objectives. Congressional demands for 
Title I program reporting and needs assessment should be met but should 
probably be done on a two-year cycle, in order to free resources for 
Title I monitoring and project rating. Monitoring and project rating need 
be done on an annual cycle, to assist decisions on project refunding and 
on allocation of technical assistance to individual projects. 

The Office of Education should act to shift Title I evaluation 
resources to systematic project classification and monitoring and to 
project rating, both to allow more effective (State) management of the 
Title I program and to aid in detecting promising projects that could 
later be subjected to more rigorous program strategy evaluation and then 
given wide dissemination. Title I evaluation should be directed increas- 
ingly toward systematic rating of the relative outputs of different 
projects with similar objectives and serving similar types of children, to 
identify which . pro j ects appear to work best in which circumstances. (Such 
comparisons might be made nationally, regionally, or within individual 
States.) The Office of Education should decide which types of projects 
are of greatest interest (for example, reading projects, early childhood 
education projects, dropout prevention projects) and then focus Federal 
evaluation resources and induce States and localities to focus their own 
evaluation resources on project rating evaluation designed to pick out the 
projects that appear to be doing the best job within each of these project 
categories . 

The Belmont system is a promising attempt at cooperative Federal-State 
data collection and analysis. 15/ Belmont has developed a number of data 
collection instruments without having a clear data analysis plan or a plan 
for use of the data, however. The Belmont Project Descriptor effort is a 



15 / See the Belmont discussion in the preceding section, which includes 
several specific conclusions and recommendations not repeated here. 
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particularly promising effort to develop a project classification system 
which, with proper input from Title I operations and technical assistance 
staffs, could become the basis for a Title I project rating system. The 
Project Descriptor effort will make it possible to automatically sort 
compensatory education projects into groups of projects with similar 
objectives, operating in similar contexts, and serving similar types of 
children. But further effort is indicated; an effort to induce projects 
to use comparable output measures as well, at least those projects that 
look promising enough to be candidates for nationwide or statewide dissemi- 
nation. The Division of Compensatory Education should attempt to have 
developed and used, as part of the Belmont Project Descriptor effort, 
’’suggested’ 1 output measures for some or all of the types of projects of 
highest priority within Title I. 

2 . Next Steps 

a . Program impact evaluation, ^program reporting, and needs 
assessment . The Division of Compensatory Education should work with the 
Belmont system to make best use of data currently available through 
Belmont and to shape Belmont development to meet Title I reporting, needs 
assessment, and program impact evaluation needs more fully. In view of 
Belmont's capabilities for producing national (and State-by-State) Title I 
program reporting and needs assessment data on both Title I and non- 
Title I schools and in view of the low priority Title I program impact 
evaluation should have at the present time, the Office of Education should 
not mount additional, independent Title I evaluation efforts of these three 
types * 



Division of Compensatory Education staff members should participate 
in Belmont working groups that will help shape (1) efforts to develop an 
"anchor test" that would allow one to equate results on different 
standardized reading tests, (2) efforts to develop and test "common status 
measures" that would be collected in addition to or in place of the output 
measures each project would normally collect, and (3) efforts to develop 
new criterion reference instruments. 

b. Program strategy evaluation . The Office of Program Planning 
and Evaluation should include in the Office of Education fiscal year 1971 
evaluation plan program strategy evaluations that examine the effects of 
concentrating compensatory education funds on smaller numbers of schools 
or smaller numbers of pupils within a school, rather than spreading them 
thin. Such program strategy evaluations might include data from Title I, 
Follow Through, and Title VIII projects. 

c. Projecc rating . The Belmont Project Descriptor working group 

and the Division of Compensatory Education should work with a small number 
of State Educational Agencies to develop and demonstrate "model" output- 
oriented project classification and rating systems that could pick out the 
top 10-25% (and the bottom 10-25%) of each group of compensatory education 
projects that have similar objectives, serve similar populations, and 
operate in similar environments. Early efforts should be made to develop 
the project rating system in high-priority Title I areas: reading, mathe- 

matics, early childhood education, dropout prevention, etc. 
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d. Project evaluation . The Division of Compensatory Education 
should alert State Educational Agencies to be on the lookout for excessive 
Local Educational Agency Title I evaluation expenditures. Since definitive 
evaluation of the effectiveness of most individual Title I projects is 
beyond present methodology or outside reasonable cost constraints, local 
evaluation of most Title I projects should normally be limited to short- 
term comparisons of project results with previously stated performance 
objectives or with baseline conditions. More elaborate data collection 
would be justified in cooperative evaluations typically comparing the 
relative outputs of projects in several school districts. 

The Office of Education, interested State Educational Agencies, and 
interested Local Educational Agencies should work together to establish 
cooperative mechanisms for more uniform, comparable, objective evaluation 
of the degree of effectiveness of different compensatory education projects 
with similar objectives. The Bureau of Elementary and Secondary Education 
and the Office of Program Planning and Evaluation should focus evaluation 
resources on demonstration projects helping groups of States or groups of 
communities serving similar populations to develop more comparable project 
evaluations (short-term and longitudinal comparisons of the results of 
different projects serving similar types of children, where the projects 
have the same general objectives but exemplify alternative treatments). 

Such efforts might be focused at first on projects nominated by State 
Educational Agencies or Local Educational Agencies as being "exemplary" 
or "possibly exemplary." Such evaluation could be mainly output-oriented 
at the start. When some projects appeared to be performing better than 
others, more intensive program strategy evaluations could be mounted in 
the next year. 

e. Monitoring and cost analysis . The Office of Education should 
support legislative and administrative action to increase the funds avail- 
able to State Educational Agencies for Title I adminis tration--in particular, 
for State monitoring of Title I projects. 

The Office of Education (in particular, the Division of Compensatory 
Education) should help State Educational Agencies to monitor their Title I 
programs more effectively by helping two-to-five States to develop and test 
prototype systems for classification and reliable monitoring of local 
Title I projects through site visits and other management techniques. 

These monitoring systems should incorporate site visits to assess project 
quality and degree of compliance with Title I regulations and guidelines, 
fiscal audits, output data (to the extent possible), and views of members 
of the community served. The Office of Program Planning and Evaluation 
should include funds for the development and demonstration of such monitor- 
ing systems in the fiscal year 1971 Office of Education evaluation plan. 

In particular, the Bureau of Elementary and Secondary Education should 
develop a prototype system for combining State Title I auditing and other 
monitoring functions, since auditing and monitoring could and should be 
mutually reinforcing functions, although auditing will usually be more time- 
consuming. The Office of Program Planning and Evaluation should include 
funds for development of a prototype monitoring-plus-auditing system in the 
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fiscal year 1971 OE evaluation plan. Such a system ought to be developed 
in cooperation with State Educational Agencies interested in using such 
systems . 

The Division of Compensatory Education and the Office of Program 
Planning and Evaluation should place particular stress on preparing systems 
for estimating local per pupil expenditures school-by-school, for use in 
(State) monitoring of the extent to which Local Educational Agencies are 
achieving comparability in local school district expenditures on Title I 
and non-Title I schools. Something between gross figures on pupil-teacher 
ratios and detailed expenditures per child is required. The Division of 
Compensatory Education should spend some time with the Belmont attribution 
manual and efforts to improve that manual, to see to what extent such gross 
estimating procedures would meet Title I needs for comparability data. 



The Office of Education should also develop an improved system for 
monitoring State Educational Agency administration and monitoring of 
Title I. Such a system should be developed by the Division of Compensatory 
Education in conjunction with the Division of State Agency Cooperation, 
the HEW Audit Agency, and the Belmont system. This system should include 
participation of Division of Compensatory Education staff members in SEA 
monitoring of local Title I projects, to spot-check State monitoring/ 
auditing efforts and get independent estimates of the extent to which 
Title I regulations and guidelines are being followed in that State. 

3 . Assignment of Evaluation Responsibilities 

The Commissioner of Education should assign Title I (and related) 
evaluation responsibilities (and corresponding evaluation staff and finan- 
cial resources) roughly as follows: 

program impact evaluation--Of f ice of Program Planning 
and Evaluation (low priority for Title I, however) 

program strategy evaluation--Belmont working groups 
(low priority for Title I, however) 

project rating--Division of Compensatory Education 
and Belmont Project Descriptor working group 

project evaluation--Division of Compensatory Education 
and Belmont Project Descriptor working group (guide- 
lines and suggestions to help State Educational 
Agencies and Local Educational Agencies improve local 
pro j ect evaluation) 

program reporting--Belmont 

needs assessment- -Belmont 
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monitoring--Division of Compensatory Education 

(monitoring State Educational Agencies and helping 
State Educational Agencies develop improved systems 
for monitoring Local Educational Agencies) 

cost analysis--Division of Compensatory Education and 
Office of Program Planning and Evaluation 

disseminating results of significant evaluations-- 
Division of Compensatory Education 

developing evaluation and monitoring methodology-- 

Belmont and Office of Program Planning and Evaluation. 



O 

ERIC 



48 



41 



III. STATE EVALUATION EFFORTS: A REVIEW AND AN ASSESSMENT 



This chapter discusses substantive tasks to be accomplished by 
State evaluation efforts; constraints on State evaluation efforts; 
types of evaluation activities most appropriate to State Educational 
Agency (SEA) efforts; and the conduct of Title I program administration, 
monitoring, and evaluation in a sample of State Educational Agencies. 

The chapter concludes by recommending a long-term strategy and immediate 
next steps to strengthen and improve State evaluation efforts. 

A. The Evaluation of Education Programs 

for the Disadvantaged: the State Role 

1. Introduction 



State Educational Agencies play a pivotal role in the 
administration of the Title I Elementary and Secondary Education program. 
Title I funds are allocated according to a predetermined formula for use 
by local school districts to support a wide range of services and 
activities for disadvantaged children. The Office of Education (OE) is 
charged with issuing regulations and guidelines on the legal uses of 
those funds and with the collection of information (largely ex post facto ) 
necessary to determine what has happened as a result of the expenditure 
of funds. Local Educational Agencies (LEA’s) have the primary responsi- 
bility for the design and implementation of programs which fall within 
the legally permissible uses of Title I funds. It remains for the SEA 
to approve all project applications and in so doing, to establish, 
modify, or expand criteria for project approval (consistent with those 
specified in the law and regulations) and to reject or cause to be 
modified any project not meeting those criteria or the purposes of 
Title I. Once projects are approved and funds are allocated, the SEA 
is responsible for monitoring local operations to see that they conform 
both with the letter and the spirit of the law and that the activities 
are, in fact, directed towards improved education for disadvantaged 
children. Finally, the SEA is responsible for submission of periodic 
reports to OE on the effectiveness of the program. 

An SEA can meet its responsibilities under Title I in any of several 
ways: (1) by assuming a passive posture vis-a-vis both the Federal 

government and local school districts within its borders; (2) by assuming 
the initiative in the direction of Title I (and other education activities 
as well); (3) or by a combination of the two. Under the first option, 
the SEA dutifully carries out whatever OE requires, while maintaining a 
healthy respect for local autonomy. In other words, the SEA acquiesces 
to LEA decisions on project design, content and coverage, so long as the 
projects proposed meet the legal requirements set down by OE. Under a 
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"passive" administration of Title I, the impetus to do evaluation is 
reduced as is the utility of the information produced by evaluation 
efforts. Information collected on program operations is intended 
primarily to meet OE needs and requirements, since the SEA exercises 
little control or direction over projects. When a State assumes a more 
"active" role in the administration of the Title I program, it focuses 
its efforts on finding ways to improve education programs. Title I is 
treated as one part of the total State education effort, and its admin- 
istration is integrated with other SEA activities. Planning and evalua- 
tion activities increase in significance as the SEA assumes more initiative 
in the direction of its educational programs. 

The active/passive dichotomy in State administration of Title I (or 
any other education program) is largely hypothetical. In reality, State 
administration of Title I can fall anywhere along a continuum from 
passive to active. The amount of direction and control over the Title I 
program exercised by a State may depend in large part on the personalities 
involved, on the politics of State education, or on the power relationships 
which exist within the State. The sections which follow center on the 
tasks which evaluation activities ought to accomplish, the constraints 
which affect the performance of those tasks, and the types of evaluation 
which are most appropriate to State efforts. 

2 . Substantive Tasks to be Accomplished 

by State Evaluation Efforts 

State evaluation efforts, like Federal and local efforts, should 
be directed at improving the effectiveness of educational programs and at 
improving the efficiency of the mechanisms which are involved in the 
delivery of those programs. Throughout this chapter, the terms 
"evaluation of Title I" and "evaluation of education programs for the 
disadvantaged" are used synonymously. At the State and particularly at 
the local level, the distinction between Title I and other Federal, State, 
or local programs for the disadvantaged becomes extremely vague. In 
evaluation, it is impossible--except by attribution--to separate out the 
effects which are due to expenditure of Title I funds from the effects of 
other local or Federal funds. 

The role of the SEA in Title I evaluation is made even more 
complicated by virtue of the fact that the SEA has its own rationale for 
existence independent of Title I. Were Title I the only program operated 
by the SEA--or even were the only Federal education program for which the 
SEA is the exclusive agent--then the complex task of running the Title I 
program would be greatly simplified. The SEA, however, raises and 
expends funds for education independently of the Federal government and 
plays a significant role in the operation of all educational activities 
within its borders, in addition to being the intermediary from Federal 
government to local school districts in operation of the Title I program. 
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As a result both of the difficulty in isolating the effects of 
Title I and of the fact that Title I itself is only one part of a multi- 
faceted SEA program, the activities carried on in the evaluation of 
Title I ought to be an integral part of the total SEA operation. The 
information generated in conjunction with the administration of Title I 
should meet both the internal needs of the SEA and the external demands 
of the Federal government, local school districts and other interested 
pressure groups within the State. Because Title I funds go into the 
majority of LEA * s in a State and because the law requires the State to 
report annually on Title I effects, Title I provides a vehicle for the 
SEA to amass potentially valuable information about the operation and 
effectiveness of education programs within its borders. Because Title I 
supports a wide variety of services and activities, the lessons learned 
from the program should be applicable to other education programs, 
regardless of source of funds or target populations served. 

Three tasks can be identified which should be included in State 
evaluation efforts in general and Title I efforts in particular. The 
extent to which a State assumes an "active" or a "passive" role in 
administering its programs will determine the extent to which each of 
those tasks is accomplished. State evaluation efforts should ideally 
be directed towards: 

-- Assisting and improving State program planning and 

management efforts . 

At the State level as well as at every other level of organization, 
there should be a link between evaluation and the planning and operation of 
programs being evaluated. If evaluation has no impact on State decision- 
making, and if State officials 1 needs for information are ignored or 
unsatisfied by the evaluation efforts, then those activities have form 
but no substance. Evaluation should be only a means towards the improve- 
ment of educational decision-making; not an end in itself. If State 
evaluation activities are directed simply at satisfying Federal reporting 
requirements or at "holding the hands" of local school districts, then 
this first task will not be accomplished. 

Evaluation activities should be designed to yield information to be 
used as inputs to overall State planning activities and as inputs to 
decisions to approve or disapprove individual projects. That is, State 
evaluation efforts ought to provide information on (1) whether existing 
Title I activities as a whole complement other State and Federal efforts 
to improve educational opportunities for the disadvantaged and (2) whether 
and how much these efforts are coordinated with existing State and Federal 
programs to yield the most effective use of limited resources. Evaluation 
should yield information which indicates the existence of gaps in activities 
and services and whether such gaps could be filled by use of existing 
State or Federal resources. Evaluation should be useful to State program 
officers in reviewing project applications by indicating which types of 
activities have successfully met Title I objectives (as defined in the 
criteria established by the State and by OE) , and which types of activities 
have had little or no impact, or a negative impact. 
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-- Assisting and improving local program planning and 
management efforts . 

The more actively a State administers its programs, the more its 
evaluation efforts will be directed toward the aim of improving local 
education by providing assistance in the design of local projects and in 
the design of procedures for the evaluation of those projects. 

The SEA can both act as a resource for local school systems and set 
criteria for local efforts. Title I provides a good vehicle for the 
SEA ? s assumption of an important role in local planning and evaluation 
efforts. The legislation gives to the State the authority to establish 
criteria for local planning and evaluation of projects and assigns 
specific responsibility to the State to determine that projects meet 
those criteria (Sec. 205 of Title I). The SEA, through its own actions, 
indicates to its LEA * s exactly how important- -or unimportant- -it consider 
evaluation. If the State undertakes a program of evaluation activities, 
but fails to use any of the findings of those evaluations to alter or 
augment the direction of its. programs ,. then the message to local school 
authorities is clear: The State Department of Education considers 

’’evaluation" something which must be done in order to satisfy some 
external requirement and not because it may be of use in improving 
education. 

State Departments of Education should work with LEA 1 s to improve 
local project evaluations. States should design and/or participate in 
evaluation activities which will help local school officials by gather- 
ing evidence on successful projects: their organization* their content, 

their cost. Similarly, each State should seek to determine which types 
of projects are least effective, so that local school systems can 
eliminate or redirect them. Through their monitoring and auditing 
activities, State agencies can provide valuable information to LEA’s on 
the experience of other school districts with similar, problems or 
projects, as well as contribute timely information from objective per- 
spective on the strengths and weaknesses of the projects under review. 

-- Assisting Federal planning and evaluation efforts . 

Regardless of the philosophical perspective from which a State 
views its role in the direction of education activities, its evaluation 
efforts will be directed toward assisting Federal planning and evaluation 
As the legal intermediary between Federal policy and local execution of 
those policies, the SEA is called upon by OE to supply information and to 
assist in the collection of information from local school districts. For 
a variety of reasons, which will be dealt with later in this chapter, 
State evaluation efforts have concentrated primarily on this third task. 
The fact that all SEA * s collect information for OE does not mean that the 
task of assisting Federal planning and evaluation efforts is done well, 
however. 
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As a result of the extensiveness of the Title I program, OE cannot 
directly monitor the operations of local projects; the States must assume 
a major role in ensuring that Title I projects and programs conform to 
Federal guidelines and regulations. Existing OE information systems rely 
on information gathered from sample surveys. While the data yielded 
through the survey technique are valuable for many purposes, such surveys 
do not relieve OE and the SEA * s of their legal obligation to ensure that 
all Title I funds are being expended for the purposes and in the manner 
intended by the legislation. The SEA has a dual function in this regard: 
it monitors the operation of Title I projects and also assists local 
districts to understand and implement the legal requirements of the 
program. 

To accomplish effectively the task of assisting Federal planning 
and evaluation efforts, the State must have an efficient monitoring system 
capable of yielding relevant information on local program operations. 
Monitoring activities should feed information up to OE and down to local 
school districts. A State which provides information useful for Federal 
planning and evaluation efforts will almost certainly be gathering infor- 
mation which will be useful in improving the planning and evaluation of 
its own programs. 

3 . Constraints on State Evaluation Efforts 

The present scope of SEA evaluation activities is defined in 
part by the constraints which operate on those activities. Some of the 
constraints are internal: that is, they arise out of prevailing State 

attitudes about the control of education or out of administrative problems 
within a particular agency. Other constraints result from forces beyond 
the control of a particular SEA. The existence of these constraints not 
only must be acknowledged, but also must be taken into account in the 
formulation of any strategy for State evaluation efforts. 

a. Methodological constraints . Throughout this study, a 

recurring plea was heard from SEA personnel: ,r We need help in evaluation." 

This most often means assistance not only in setting up the mechanisms for 
data collection and processing, but also assistance in selecting and 
applying appropriate evaluation tools. Methodology has been one of the 
weakest links in evaluation at every level: knowing what to measure and 

knowing how and what conclusions to draw are problems which plague evalua- 
tions of all social programs. Educational evaluation at the State level 

is hampered by a lack of specificity in project and program objectives, 
inadequate or ill-defined output measures, and inability to describe 
process variables simply and inexpensively. This condition will be less 
of a constraint on evaluation as better techniques of evaluation are 
developed, tested, and disseminated. 

b. Political constraints . Public education in the U. S. has 
been described as the last of "the cottage industries." Local control of 
education has been canonized and is an operative principle in American 
education. As a result, SEA’s have played a traditionally passive role 
in the direction of education. SEA personnel typically cajole, suggest, 
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urge and perhaps even exhort--but almost never REQUIRE--LEA 1 s to make 
appropriate changes in educational practice. 

Moreover, public education until very recently has not been held 
accountable by the community for the services it provides to children. 

This lack of accountability for success or failure of education at all 
levels of the governmental structure has important implications for 
evaluation. The absence of public pressure for accountability removes 
one of the key incentives for evaluation. If what works, or why, seems 
unimportant, there is no incentive either to evaluate or to use the 
results of evaluation in policy formulation. Evaluation becomes an 
academic exercise . 

At the moment, there is very little incentive from the SEA for local 
evaluation. Projects are funded with or without evaluation components 
and in most instances, irrespective of evaluation results. Similarly, OE 
provides little incentive to the SEA's to do evaluation or to improve 
evaluation. For the past three years , OE has required the States to file 
annual evaluation reports based on a set of broadly worded questions, but 
to date, the States have seen no evidence that OE has ever even used the 
materials supplied by those State reports. The quality and quantity of 
evaluation activities will increase if and when evaluation is demonstrated 
to affect decision-making. 

c. Manpower constraints . There are not enough people working 
in evaluation at the State level and possibly not enough of the right 
kinds of people. State evaluation efforts are badly understaffed either 
because the SEA has not allocated sufficient positions for evaluation 
(which is in part a reflection of how the SEA views the utility of evalua- 
tion) or because the positions which are allocated go unfilled. Salaries 
paid to State evaluation personnel (and other State personnel as well) 
are often not competitive either with those paid by large urban or 
suburban school districts or with the salaries a qualified evaluation 
specialist could command elsewhere in the marketplace. Positions also 
go unfilled because of complicated State hiring procedures and irrelevant 
job qualifications. The intricacies of SEA administration have been major 
contributors to the anomalous situation in which SEA's have in the past 
returned unused administrative monies to the U. S. Treasury (under both 
Titles I and III) despite the need for more evaluation staff. In many 
instances, the people who are responsible for evaluation lack formal 
training in some of the quantitative skills which are tools of the trade. 
Evaluation staff are often recruited through traditional and restrictive 
channels; classroom teaching, local school administration or laterally, 
from other jobs within the SEA. 

Amending State personnel practices or salary structures is a long- 
range objective, but much can be done in the short-run to upgrade the 
skills of existing State personnel involved in evaluation. For example, 
in-service training materials and workshops on the purposes, design, and 
utility of evaluation can and should be developed; cooperative efforts 
between SEA's and nearby colleges or universities can be developed for 
exchange of staff or to augment SEA staff on a part-time basis. 



54 



47 



d. Resource constraints . Evaluation is expensive. Although the 
amount of money going into evaluation has increased over the past few years, 
there is still not enough money for evaluation from either Federal or State 
sources. Titles I, III, and V have been the principal sources of Federal 
support for State evaluation activities. Beginning in FY '70, ESEA 
Section 402 funds were added to this list. But in every case, the Federal 
funds have been spread across a variety of State agency activities, only 
some of which are directly related to evaluation. 

According to the fifth annual report of the Advisory Council on State 
Departments of Education, only 8.75% of the total expenditures of SEA’s 
(in FY *69) were spent for all planning, development, and evaluation- • '■ 

activities. Like Federal funds, these State funds are used to support a 
variety of activities. Site visits conducted during this study indicate 
that a large part of the evaluation resources which SEA 1 s use goes into 
data collection and processing activities and into support for the tradi- 
tional fiscal audit activities. 

e. Scheduling constraints . Up to this time, evaluations have 
tended to culminate in the production of documents submitted on an annual 
cycle. However, there is little if any congruence between the cycle of 
evaluation and either program planning or decisions concerning funding. 
Evaluations are typically submitted in the form of annual reports from 
LEA’s to SEA’s. The reports for one school year are received by the SEA, 
however, after decisions have been made concerning programs for the next 
school year. The same holds true for the reports supplied by SEA’s to OE 
and for the little survey information which OE sends back to SEA’s. Little 
if any effort is being made by SEA’s to collect periodic evaluative infor- 
mation which might be useful in the decision-making process. Evaluation 
systems will have to be designed which include the traditional evaluation 
reports, but which also include provision for the generation of more timely 
data, perhaps on a quarterly basis. 

4 . Types of Evaluation Activities 

Most Appropriate to State Efforts 

In Appendix B of this report, we identified eight types of 
evaluation and evaluation-related activities: program impact evaluation, 

program strategy evaluation, project evaluation, project rating, monitor- 
ing, program reporting, needs assessment, and comparative cost analysis. 

Some of these evaluation activities are more appropriate for SEA’s than 
others, in terms of Title I and similar programs. The appropriateness of 
each type of evaluation must be viewed not only in terms of what is desir- 
able, but also in terms of what is practical, given the constraints which 
were discussed in the preceding section and the present state of SEA 
evaluation efforts . 

a. Program impact evaluation . It is clearly appropriate for a 
State to attempt to determine the overall effectiveness of all its programs 
for the disadvantated , but such program impact evaluation is extremely 
difficult and costly. Program impact evaluation is meaningful only if the 
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SEA has defined the objectives for such programs and has selected appro- 
priate measures to assess progress toward those objectives. Pennsylvania's 
Quality Assessment Project is an example of program impact evaluation. 

Up to this point, the Pennsylvania effort has been primarily a research 
activity; this fall it will enter its operational phase; therefore, no 
conclusions have been reached regarding overall program effectiveness. 

At times in the past, program impact evaluation has incorrectly been made 
synonymous with the sum of the effectiveness of individual local projects 
within a program. Since the effects of Title I cannot be separated from 
those of other State or local programs and since Title I local project 
evaluations are not consistent in design or measures used, statements 
made about the impact of Title I based on local project evaluation must 
be greatly discounted. Given the scarcity of funds for evaluation and 
the present state-of-the-art in evaluation, States should be reluctant to 
invest in Title I program impact evaluation. 

b. Program strategy evaluation . It is important for State 
personnel to know about the relative effectiveness of various education 
strategies and, therefore, program strategy evaluation is appropriate to 
SEA needs. However, program strategy evaluation may not be feasible for 
all SEA's for several reasons. Such evaluations are expensive to carry 
out and would probably exceed the current ability of any one State to pay 
the costs. Moreover, many States do not have within their borders suffi- 
cient variation among program strategies or sufficient examples of each 
strategy necessary for this type of evaluation. Neither of these factors 
pose insurmountable difficulties. As resources--in terms both of money 
and manpower--grow, the ability of a State to support program strategy 
evaluation will increase. As a State improves its information about what 
is happening in its local districts, it will improve its potential for 
finding the necessary examples of program variation or can take the lead 

in inducing LEA's to adopt various program strategies as part of an overall 
evaluation effort. Some States may wish to enter into cooperative program 
strategy evaluations with neighboring States. 

The Office of Education should facilitate such cooperative State evalu- 
ation efforts o OE should provide assistance upon request in the design and 
execution of program strategy evaluation. More importantly, OE can itself 
design and fund program strategy evaluation. In doing so, OE ought to 
seek the active involvement and participation of SEA's. Results of program 
strategy evaluations will clearly be applicable to more than one SEA. 

Program strategy evaluation will yield crucial information about which 
types of projects are most effective in meeting the educational needs of 
disadvantaged children. Utilization by a State of the results of such 
evaluations will certainly move the State toward "active” administration 
of its programs. 

c. Project evaluation . Not only is project evaluation an 
appropriate activity for the SEA, but it is the most common type of 
evaluation extant today. Local project evaluations currently yield 
little useful data relative to the money and manpower invested, however, 
primarily because the evaluations are noncomparable. The SEA can play 

a critical role in changing this situation. 
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First, the State can improve local project evaluation by providing 
direction and leadership to LEA ! s on what to evaluate and how to do it. 

The SEA should provide explicit instructions on the design of project 
evaluations, offer examples of appropriate techniques or instruments, 
recommend and/or locate consultants, and set guidelines for expenditures 
for such evaluations. Some States have already begun to take steps 
necessary to improve local project evaluation. For example, New York 
State has compiled an "Assessment and Evaluation Handbook" and an 
'Evaluation Manual" (for its Urban Education Program) . 

This approach of working to improve local project evaluation 
emphasizes the fact that the SEA should be interested in knowing how 
well local projects measure up to local objectives and what kind of 
progress students are making. The SEA should encourage exemplary 
project evaluation. State staff should review project evaluation 
reports, pick out examples of "good" evaluations, and disseminate 
them to LEA f s throughout the State. More importantly, the SEA should 
go beyond providing assistance on how to improve local evaluation and 
should use the results of project evaluation in its deliberations on 
applications for funds. Massachusetts claims that it will do this for 
summer project applications beginning in FY 1971. 

A second approach which the SEA can use to strengthen project 
evaluation involves the State in the evaluation directly. Once the SEA 
has the capability to identify classes of similar projects, it can use 
its own staff to conduct comparative evaluations of similar projects. 

These State evaluations should use some common measures of project 
success, either agreed on jointly by the SEA and LEA's involved or 
imposed by the SEA. In the latter case, the SEA could specify as a 
condition for funding that the LEA include one or two common output 
measures in its project evaluation. (These would be in addition to any 
other measures the LEA might choose to adopt.) Projects could then be 
compared on the basis of common outputs; comparisons could be made 
between progress on local output measures and on common State-imposed 
output measures. 1/ 

Pennsylvania has established a precedent for SEA evaluation of 
Title I projects with its on-site evaluation system, discussed below. 

SEA evaluation of local projects may be even more feasible in project 
grant programs, such as Title III. 

d. Project rating . Potentially, project rating is both useful 
and appropriate for State evaluation, but at present only parts of a 
project rating system are readily adaptable to State information needs. 

A fully developed project rating system attempts to rank projects on the 
basis of their success in meeting some common objectives. A rating system 



1/ This approach is similar to project rating and could be considered as 
such if the form and scale included all projects in a given class 
(see below). 
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does not explain what causes the observed results, but it does lay the 
groundwork for the design of more intensive evaluations. Most importantly, 
a rating system locates those projects serving similar populations which 
are most successful in meeting a set of objectives, and those which are 
least successful. Such information would be useful to State officials and 
to local officials in the design and operation of compensatory projects. 

There are several difficulties which must be overcome before a State 
can install a project rating system. First, a project rating system 
depends on the specification and collection of comparable output data. 

If an SEA can induce some LEA 1 s to report such comparable information for 
projects serving similar populations, then it would be possible to implement 
a rating system. Furthermore, a rating system groups projects into classes 
based on environmental variables (such as student SES) or on inputs. The 
projects in each class would be expected a priori to yield similar results. 
Unless the SEA collects sufficient uniform environmental data and input 
data on projects operating in its LEA f s, it will not be able to develop the 
classifications within which projects are rated. 

SEA's can develop a rough classification scheme for projects based on 
information collected annually in project applications. These data can be 
used to classify projects, or the forms can be modified with relative ease 
to yield the desired information. SEA's should be encouraged to establish 
such classification schemes at least as a basis for more systematic 
monitoring activities. 

e. Monitoring . At present, monitoring is the most prevalent 
type of evaluation activity in SEA ! s. Information collected during the 
site visits and discussions with State officials indicate that most States 
fail to exploit the potential usefulness of monitoring activities to 
accomplish the substantive tasks described in Section A-2. Current 
monitoring efforts too often are cursory and non-sys tematic . Often, 

States monitor by exception: for example, Texas which looks into projects 
which are, have been, or are likely to be in trouble, or New Jersey which 
concentrates systematic monitoring only on the large LEA's. No State 
appears to have a system for monitoring projects or LEA 1 s both in terms 
(1) of whether the project conforms to its application and to the legal 
requirements; and (2) of how well the project is doing vis-a-vis others 
having similar objectives and serving similar populations. Because 
subjective judgments are made by persons doing monitoring, the latter 
type of monitoring does take place occasionally and informally. However, 
to make monitoring more objective, a classification system should be 
developed for all projects. 

Monitoring provides the opportunity for the State to gather informa- 
tion on what is actually happening. It is clearly a necessary means of 
determining whether projects conform to legal requirements; moreover, 
monitoring visits provide a valuable opportunity to offer technical 
assistance to LEA 1 s on program operations. Monitoring presently is done 
by a variety of State personnel: program officers, auditors, consultants, 

and curriculum specialists. Apparently, there is little systematic train- 
ing of on-site monitors. Monitoring instruments (used in writing site 
reports) tend to emphasize subjective judgments or call for responses to 
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vaguely worded questions about project operations. On-site reviews con- 
ducted by State officials would benefit greatly from (1) the development 
of more objective means to monitor projects, (2) more rigorous training 
of the monitors prior to the site visits, and (3) systematic follow-up 
procedures to see if recommended changes are made. 

There is probably no single monitoring system which will meet the 
needs or capabilities of all 50 States. The issues on which monitoring 
activities will focus may vary from State to State: for some, the primary 

issues will relate to fiscal disparities within the LEA; for others, the 
central issues will be selection of program participants; for others, 
development of comprehensive programs or more effective sequencing of 
services. The techniques for gathering monitoring information may also 
vary depending on the capacity and budget of the SEA, the number of LEA 1 s , 
etc. As a result, some monitoring systems may rely primarily on on-site 
visits, while others will seek to blend on-site visits with information 
derived from other sources within the SEA or through written or telephone 
contacts with the LEA's. OE, and DCE in particular, ought to work closely 
with several States to develop and demonstrate monitoring systems for 
Title I projects. These prototype monitoring systems should concentrate 
on developing a variety of techniques for gathering the necessary informa- 
tion on project operations quickly and cheaply. The monitoring systems 
should attempt to take existing or easily obtainable information and 
assemble it in a manner which can be used by State program administrators 
and which can be fed back to LEA 1 s . 

f. Program reporting/ needs assessment . These are absolutely 
essential activities for SEA f s. Like the Office of Education, SEA's tend 
to have incomplete program statistics on populations served, services 
provided, staffing patterns, program expenditures, etc. Data on these 
facets of project or LEA operations have tended to come almost exclusively 
from annual project applications. State program officials should also 
collect information on educational needs in order to determine where gaps 
exist . 



Parts of the Belmont system (specifically, data from the Consolidated 
Program Information Report and the Pupil-Centered Instruments) contain 
information about needs and services provided from a sample of LEA's 
which is representative for each State. To date, information collected 
through the Belmont system has not been provided to States on a timely 
basis or in a form that is useful to the States. If this situation can 
be corrected, then States will have a tool which can be used in a variety 
of ways. If Belmont cannot be made more responsive to State information 
needs and more timely, then States will have to expend some of their 
limited resources to collect such information. 

g. Comparative cost analysis . Cost data should be a part of 
the regularly collected program information. To the extent that these 
data are reliable, they should be used in making comparative analyses 
of costs wherever useful to the SEA planning and management efforts. If 
accurate cost data are not readily available (or some reasonable approxi- 
mations thereof), limited SEA evaluation resources should not be expended 
to get those data. 
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In conclusion, all eight evaluation act ivit ies are more or less 
appropriate for States to undertake, but at this point in time, monitoring 
and other activities related to it should be emphasized. Monitoring can 
and should become the point at which aspects of a project rating system, 
improved program reporting/needs assessment, and the results of local 
project evaluation are brought to bear on the improvement of local pro- 
grams. When this happens, a monitoring system provides relevant and timely 
information and becomes an effective tool for State program administrators. 
Until monitoring is done well, both program impact and program strategy 
evaluation will remain luxuries that States can ill afford. 
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B. The Conduct of Evaluation Activities in 
Selected State Departments of Education 

1 . Introduction 



The roles that State Departments of Education play in the 
administration of public education are far from uniform. In the past, 

SEA's have suffered from inadequate financial support and severe diffi- 
culties in recruiting qualified personnel. For a variety of reasons, 

State Departments of Education have tended to focus on the problems of 
rural or small school districts, while urban and suburban systems were 
left to go their own way. 

With the enactment of ESEA came greater emphasis on the role of 
State Departments of Education in the administration and operation of 
education programs. Title I requires SEA 1 s to review and approve appli- 
cations from local school districts; Title II operates under a State plan 
for the disbursement of funds; Title III is now largely a State-operated 
project grant program. Other pieces of Federal legislation require the 
State to assume responsibility for program operation. The added responsi- 
bilities placed on SEA's have been accompanied by increased Federal 
support for the operation of the State agencies. Title V and Vocational 
Education represent the largest sources of funds for the support of SEA's, 
but funds are earmarked specifically for general program administration 
under several other programs, such as Title I and Title III. 2 _/ 

The Federal share of total State Educational Agencies expenditures in 
FY 1969 amounted to $108 million, or 417> of the total. 3/ The Advisory 
Council on State Departments of Education reports that SEA expenditures 
support a variety of activities, as indicated in Table III-l. Unfortunate- 
ly, the Advisory Council report does not explain what activities are 
included under each of the program functions. Table III-l does show that 
"Improvement of Instruction," which is the traditional activity of State 
Departments of Education, accounts for $2 of every $5 spent. 

The amount of total SEA expenditures which is accounted for by Federal 
funds varies from State to State, as does the amount spent in the category 
called "planning, development, and evaluation." The Federal share of total 
SEA expenditures in FY '69 varied from a high of 69.67, in Texas to a low of 
12.07, in Virginia. Table III-2 illustrates the amount of Federal support 
and the amount spent on planning, development, and evaluation in the States 
visited during this study (plus New York and California). 

Table III-2 also illustrates the variation which exists among the 
States. New York and California together account for over 207c of the total 
expenditures by all SEA's. The Federal share of expenditures in these eight 



2 / For a State-by-State breakdown of Federal funds available for State 
administration, see an HEW internal paper by Gilbert Austin, "The 
Federal-State Partnership in Education," dated June 1, 1970. 

3/ U. S. Department of Health, Education, and Welfare, The Federal-State 
Partnership for Education , Washington: March 1970, p. 15. 
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TABLE III-l 



Total Expenditures of All State Educational Agencies 
by Program Function, FY 1969 a/ 





Function 


Amount 

(in millions) 


% of Total 


1 . 


General Administration 


$ 57.7 


21.9 


2. 


Planning , Development , 
and Evaluation 


22.9 


8.7 


3. 


Improvement of Instruction 


103.1 


39.2 


4. 


Improvement of Administration 


22.5 


8.5 


5. 


Accreditation, Licens ing , 
and Staff Development 


18.3 


6.9 


6 . 


Agency Operated Programs 


24.3 


9.2 


7. 


Other 


13.5 


5.1 




Total 


$262.4 


100.07, 



a/ U. S. Department of Health, Education, and Welfare, The Federal-State 
Partnership for Education , Washington: March 1970, p. 15. 
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(largely urban) States vary from 70% in Texas to 22% in New York. The 
expenditure for "planning, development, and evaluation" exceeds the 
national average in five of the eight States, ranging from a high of 13% 
in Pennsylvania to a low of 1% in Massachusetts. 

The following sections of this paper will examine "evaluation" 
activities in the five States visited, in an attempt to understand what 
the term means in practice, what types of staff and resources are used 
in evaluation, and what might be done to improve State evaluation efforts. 

2 . A Review of Activities in a Sample of States 

During the month of June, staff from the Urban Institute visited 
five State Departments of Education: Massachusetts, New Jersey, Oregon, 

Pennsylvania, and Texas. [A much shorter visit to Ohio was made in late 
July.] The sample, which was selected after consultation with Office of 
Education staff, was intended to be representative of evaluation and 
technical assistance efforts in all 50 States. The site visits were not 
intended to review all aspects of SEA administration, or even of all 
Federal programs. Interviews focused on State administration of the 
Title I program with particular emphasis on four areas: project applica- 

tion, monitoring, evaluation, and technical assistance. While site visits 
concentrated on Title I, some information was gathered on other facets 
of SEA operations, such as the Title III program, Title V "flow-through" 
funds, SEA research and/or central planning activities, fiscal audit 
procedures . 

The site visits were designed to get a better picture of how State 
Departments of Education attempt to improve the operation of the Title I 
program in particular and education of the disadvantaged in general. 

Through interviews with State personnel and review of available documents, 
an attempt was made to determine the extent to which evaluation-related 
activities supported the substantive tasks outlined in the first part of 
this chapter. Underlying the discussions with State personnel was the 
assumption that information generated in the name of evaluation or program 
administration ought to be of use primarily in meeting the needs of State 
officials . 

Discussions relating to the area of program administrat ion/operations 
focused on the organization, staffing patterns, and assignment of responsi- 
bilities for the administration of the SEA as a whole and of Title I in 
particular. More importantly, the procedures used in the annual applica- 
tion, review, and approval process of the Title I program were examined in 
detail because of their importance as decision points in the program cycle. 
The project application process is the point at which the results of 
evaluation and monitoring activities ought to impact the direction of the 
program. Our concern with monitoring centered on the procedures and 
activities used by the SEA to find out what was going on in a particular 
program, especially to determine if the projects were being operated in 
accordance with prescribed objectives, regulations, and guidelines. 
Interviews about evaluation attempted to identify evaluation activities 
being conducted by the State, the manner in which the State fulfilled its 
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mandated requirements for evaluation, the extent of State use of past 
evaluations, and the impact of evaluation results on the direction of the 
program. Finally, the activities of the SEA called ’’technical assistance" 
were investigated. The role of States in technical assistance will be 
discussed in Chapter VIII of this report. 

a. Program organization /administration . The organizational 
patterns within the five State Departments of Education visited and the 
positions of the Title I programs and staffs within those organizations 
vary significantly. SEA’s appear to be in the midst of a reorganization 
which generally involves more centralization of planning and evaluation 
activities. Both Oregon and New Jersey have recently 1 reorganized; Texas 
has had a system of consolidated program administration for several years; 
Massachusetts is considering using part of its ESEA 402 money to create 
a central planning staff in the Office of the Commissioner, which 
Pennsylvania has already done. 



Similarly, there are important differences in the way the Title I 
program is operated in the five States. New Jersey, which has 450 school 
districts, concentrates most of its Title I administrative efforts on the 
26 largest LEA’s (which account for about 907 o of the funds) and places 
particular stress on improving the management of Title I projects. To 
accomplish this, the large LEA’s in New Jersey are required to develop 
and adhere to PERT systems for their Title I operations. The remaining 
424 LEA’s work with 21 county superintendents, who are SEA employees 
reporting to the Deputy Commissioner. The county superintendents, who 
have their own staffs, spend part of their time screening Title I project 
applications from small LEA’s and occasionally work with them on other 
aspects of Title I operations. 

Like New Jersey, Pennsylvania has decentralized part of its adminis- 
tration of Title I. The State has established 16 Education Development 
Centers located in State colleges or universities and headed by a college 
professor who works part-time for the EDC. Eight of the EDC 1 s have full- 
time assistant directors, who work primarily on Title I projects in the 
LEA’s within the region served by the EDC. 

Massachusetts , which has roughly the same number of local districts 
as New Jersey, tries to spread its central Title I staff over all LEA’s. 

The State does have four regional offices, but regional staff are used 
only sparingly in the operation of the Title I program. In Massachusetts, 
each Title I staff member has responsibility for all aspects of the Title I 
program in all LEA’s within a particular geographic area of the State; the 
State Title I coordinator is responsible for the five largest LEA’s. 

Oregon makes no attempt to divide Title I project applications on the 
basis of location within the State, project content, or staff expertise. 

In fact, applications are assigned for review and approval on a first come- 
first served basis. 
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New Jersey, Massachusetts, Oregon, and Texas rely heavily on OE 
guidelines and instructions in reviewing Title I project applications. 
(Texas places particular emphasis on the issues of concentration, 
comparability and the necessity to eliminate instances where Title I 
might be used as general aid.) Pennsylvania was the only one of the 
five States visited which is requiring that, in project applications, 
the LEA 1 s must state their instructional objectives in measurable terms. 

Beginning in FY *71, Pennsylvania will accept two-year applications 
for Title I projects to enable local districts to do more planning 
(initial approval of a project will not ensure automatic approval for an 
extension into the second year.) In Texas, the Title I program is one part 
of the Division of Program Funds Management, which is supposed to receive 
and review consolidated applications for seven Federal programs, including 
Title I. Although the consolidated application is intended to stimulate 
comprehensive local planning, Texas Education Agency staff readily admit 
that comprehensive planning has not yet advanced much beyond the stapling 
together of what used to be separate applications for Federal funds. 

While the method of reviewing and approving project applications 
differs from State to State, some observations can be made concerning 
all five States. In every case observed, the project application review 
process was less than systematic. To date, none of the States visited 
has exploited the potential usefulness of information contained in 
project applications as the basis for monitoring. There appears to be 
very little feedback of information to LEA's within the State. For 
example, none of the States regularly ^produces a listing of the LEA's 
which operate projects of similar size, scope, and content# 



b. Monitoring . Even though monitoring activities in the five 
States vary in focus and process, site visits form the keystone of State 
monitoring. In Oregon, Massachusetts, and Texas, site visits are usually 
made by a single member of the SEA Title I staff and tend to be about one- 
half day to two days in length. These three States use the same staff who 
review and approve applications to monitor Title I projects. Both 
Massachusetts and Oregon aim at making one site visit per year to each LEA; 
in practice, the monitoring cycle is nearer one visit every two years. The 
site visit form used in Massachusetts relies on subjective appraisals by the 
monitor of the quality of the project and the extent to which the project 
is operating as described in the application. Occasionally, Massachusetts 
does use other than Title I staff from the SEA to make site visits. Oregon 
site visit reports focus on descriptive information on project operations 
and call for an overall rating of the project by the monitor. Oregon person- 
nel, however, could supply no data on the proportion of projects which fall 
into each of their four rating categories. 



Texas concentrates its monitoring activities on LEA's where problems 
have been uncovered either during review of the project applications or 
during the preceding year. Thus, in Texas, monitoring is tied directly 
to the application process. Issues of concentration and comparability are 
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of primary concern in Texas site visits , while Massachusetts and Oregon 
staffs focus on program content. Local Title I expenditures in Texas are 
monitored through quarterly financial reports submitted to the Program 
Funds Management staff, while fiscal audits (which are on a five-year 
cycle) are handled by another part of the Texas Education Agency. In 
Oregon and Massachusetts, the auditor is a part of the Title I staff, 
although in Oregon site visits play a relatively small part in the audit 
process . 

Monitoring procedures in New Jersey and Pennsylvania are different 
from those used in the above three States. Both States utilize a team 
approach to site visits. New Jersey monitors by exception. Each of the 
26 large LEA’s is required to file monthly program evaluation reports 
indicating the current status of Title I projects. The Title I project 
directors rate the progress of their projects and cite any problems which 
may have arisen. If a problem is critical, the SEA sends in a monitoring 
team; if there is no crisis, the monthly report is attached to the LEA 
file for use on regularly scheduled site visits. Each of the large LEA's 
is visited by a monitoring team at least once and probably several times 
per year. Written as well as verbal reports are given to the LEA's. 
Beginning in FY *71, the Title I staff will concentrate monitoring efforts 
on program management, while other SEA staff, i.e., evaluation staff and 
subject matter specialists, will assume responsibility for monitoring 
program content. 

Pennsylvania has an on-site evaluation or monitoring system which 
uses monitoring teams assembled by the SEA from among a list of college 
teachers, principals, local Title I directors, and teachers. These 
evaluators, who must come from outside the LEA being monitored, are paid 
from local Title I funds which the SEA requires the LEA to set aside for 
that purpose. In school year 1969-1970, Pennsylvania operated on a two- 
year cycle of visits, but is shifting to a three-year cycle. The monitor- 
ing process begins with an LEA self-evaluation, which must be completed 
three weeks prior to the site visit. The monitoring team then scores the 
project on the same points covered in the self-evaluation. State officials 
view this system as useful both as a learning experience for the monitors 
and for the LEA undergoing the self-evaluation and on-site review. There 
are weaknesses in Pennsylvania’s on-site system as presently constituted: 
not enough training for monitors is provided; there are severe problems 
of reliability and validity; the connection to fiscal audits is virtually 
nonexistent; and not enough attention is paid to outputs. (In Pennsylvania, 
only 24 fiscal audits were completed last year: an apparent 20-year cycle.) 

Not one of the States visited has an adequate monitoring system, 
although each has something of value. None of the States has an adequate 
system for ensuring that deficiencies found through monitoring are 
corrected. Each SEA could benefit from a more systematic approach to 
monitoring; most could benefit from closer coordination with fiscal audits 
and with other parts of the SEA involved in site visits. There is probably 
no single monitoring system which would be applicable in every State; 
therefore, several different models ought to be developed. Because of the 
variation which exists among the States in the way they monitor programs, 
all States could likely benefit from sharing information about existing 
monitoring procedures &nd ways to improve upon them. 
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c. Evaluation . Aside from the monitoring activities described 
above, there was very little in the way of evaluation going on in the five 
States visited. Each year, these States--like the other 45--are required 
to submit an annual evaluation report on the Title I program to OE. For 
the past three years, OE has given the States a set of open-ended questions 
around which to build the annual report on Title I. By and large, the 
States visited take those questions or other materials developed by OE and 
pass them on to local school districts for completion, compile the responses, 
and use them as the basis of the report sent to OE . 



The procedures used by the States to meet the OE reporting require- 
ment vary somewhat due in part to the organization of the SEA, in part to 
the pressures on the Title I staff, and in part to the importance placed 
by the State personnel on the utility of evaluation. In Oregon, even 
though an evaluation specialist is assigned to the Title I staff, the 
evaluation data collected in the State are primarily the basic data needed 
to meet OE reporting requirements. The State Department officials are 
aware of significant variation in both the quality and quantity of local 
project evaluation within the State. (Oregon officials oppose the notion 
of comparative local evaluation on the grounds that each project or LEA 
should develop individualized objectives. Project personnel are 
encouraged to visit other projects to observe and possibly adapt what is 
seen, but not necessarily to agree on common goals, objectives, and eval- 
uation strategy.) 

The Title I staff in Massachusetts are collectively responsible for 
all aspects of Title I, including evaluation. In the early years of 
Title I, the compiling of the State’s annual evaluation report was done 
on a part-time basis by several members of the already overburdened 
Title I staff. Since FY 1968, the State has relied primarily on outside 
consultants to prepare the annual report. The consultants work only 
part-time for the State and provide some technical assistance to local 
school districts in addition to the preparation of the annual report. 
Massachusetts officials are not satisfied either with the quality of the 
report submitted to OE or with the state of evaluation at the local level. 
In fact, Massachusetts recognizes that the report filed with OE is hardly 
an "evaluation" of Title I activities in the State and has dropped the 
word from the title of the report. Massachusetts plans to improve the 
quality of local evaluation by requiring that Title I projects provide 
evidence of evaluative success prior to approval by the State. However, 
because of the incongruence of the evaluation and project approval cycles, 
this requirement will initially apply only to summer project applications. 
Early this spring, the Massachusetts Advisory Council on Education issued 
a report on the operation of compensatory education programs in the State. 
Entitled Blueprint for Action , 4/ the report makes specific recommendations 
on ways to improve several aspects of program operations, including evalua- 
tion at State and local levels. 



4V Daniel G. Jordan, Blueprint for Action , University of Massachusetts 
School of Education, Amherst: March 1970. 
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The New Jersey annual report, like the Oregon and Massachusetts 
reports, is based entirely on the LEA responses to the OE questions. The 
26 largest LEA's are required to submit additional data (such as test 
scores or the results of evaluations done by consultants), but the State 
does not make use of this information in its annual Title I report to OE. 
Beginning this year. Title I evaluation activities in New Jersey are to 
be centralized. The director of the new Office of Evaluation has been 
working with the 26 LEA * s in an attempt to set up a cooperative evaluation 
effort based on collection of comparable cognitive data. The LEA 1 s have 
been asked to turn back to the SEA 1 % of their total Title I allocation 
to be used for common testing of students in the districts. While the 
LEA's appear receptive to the idea of some cooperative evaluation, there 
is resistance to comparisons of projects across districts. 

Pennsylvania, like New Jersey, has centralized evaluation. But 
unlike the other States discussed so far, Pennsylvania has invested its 
own funds in the development of a comprehensive system for evaluating 
elementary and secondary education which is independent of Title I eval- 
uation. (As in other States visited, Pennsylvania's annual report on 
Title I is based primarily on information collected from the LEA's to meet 
OE requirements.) Pennsylvania is spending about $225,000 per year on its 
Quality Assessment Project, which is an effort to establish the goals of 
quality education in Pennsylvania and to measure progress towards those 
goals. This project has just emerged from its research phase into its 
operational phase, in which school districts participate on a voluntary 
basis. This second phase of the project will gather information on all 
children in grades five and eleven of schools in the sample (about 107o 
of the population of fifth and eleventh grade students in the State) . 
Eventually, other grades will be brought into the project which is 
designed to "provide each school district with relevant comparative data 
to enable directors and administrators to more readily appraise the 
educational performance and to effectuate without delay the strengthening 
of the district's educational program. Tests developed ... [in conjunction 
with this project] shall be used for the purpose of providing a uniform 
evaluation of each school district ...."5/ If the Quality Assessment 
Project is successful, Pennsylvania will have information which can be 
used to develop a project rating system and to design program strategy 
evaluations as well. 

Texas has taken a different approach from other States visited in 
satisfying its mandated reporting requirement. Planning and evaluation 
activities in the Texas Education Agency have been centralized unde£ an 
Associate Commissioner of Education for three years and, as a result, are 
entirely removed from the operation of the Title I program. Beginning in 
FY 1970, Texas built its Title I evaluation around the Belmont system. 

Both the CPIR and the PCI have been expanded into a statistically valid 
sample of the State's 1100 school districts. In addition, the State plans 



5/ Educational Testing Service, A Plan for Evaluating the Quality of 

Educational Programs in Pennsylvania , Princeton: June 30, 1965, p. 7. 
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to collect some basic census-type data for all the Texas school systems. 

The evaluation staff in Texas expects the information generated in this 
way to provide sufficient material for the State's FY 1970 report to OE. 

The success of this approach depends largely on the quality of the OE 
analysis of the Belmont data since Texas intends to use the OE analysis 
as the basis for its own analysis. In FY 1969, Texas attempted to carry 
out an ex post facto longitudinal evaluation of Title I in a sample of 
school districts, but had to abandon the project because the local 
districts did not have the records necessary to carry out such an evaluation. 

Texas was the only State visited where monitoring and evaluation 
activities are entirely independent of one another. The division which 
does most of the site monitoring and which is responsible for Title I 
project approval plays little or no part in the evaluation process. 
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C. Concluding Observations Concerning the Organization 

and Conduct of Evaluation at the State Level 

At the beginning of this chapter, three substantive tasks were 
identified toward which State evaluation activities should be directed: 

-- Assisting and improving State program planning and 
management efforts . 

-- Assisting and improving local program planning and 
management efforts . 

Assisting Federal planning and evaluation efforts. 

Evaluation as a part of the decision-making process is a relatively 
new phenomenon at the State level. In many cases, the impetus for evalua- 
tion came exclusively from Federal requirements and specifically, from the 
requirements of ESEA. If the sample of States visited during this study 
is representative of activities among the 50 States, then it is quite clear 
that Federal requirements will continue to shape and to dominate State eval- 
uation activities. Unfortunately, even though States expend time and effort 
to meet 0E evaluation requirements, the information so generated is rarely 
useful to the States and only slightly more useful to 0E. 

During the first two years of the Title I program, the Office of 
Education prepared a set of detailed questions concerning the operation 
of the Title I program which served as the basis of each State* s annual 
report. Federal Title I officials made a conscious decision not to enforce 
uniform reporting requirements on the States, but to rely on the informa- 
tion contained in the State reports and hope that such information would be 
comparable. As a result, the first two reports to Congress on Title I 
suffered from a lack of comparable data on students, teachers, programs 
offered, or benefits associated with participation in compensatory educa- 
tion programs. Beginning in FY 1968, 0E changed its approach to Title I 
evaluation and carried out directly a national survey of selected elemen- 
tary grades to gather information on compensatory education services and 
programs. [This survey, which is now part of the Belmont system, is 
discussed in Chapter II and Appendix E.] In order to supplement the 
information in the survey, 0E revised the rather detailed set of questions 
used in the preparation of State annual reports. Since FY 1968, the 
requirements for State annual reports have taken the form of between nine 
and thirteen open-ended questions about Title I activities. These broadly- 
framed questions were intended to remove some of the more burdensome 
requirements on the States and to allow the States the flexibility to 
develop evaluation systems more nearly suited to State needs. This does 
not yet appear to have happened to any significant degree, however. 

Typically, States send out evaluation guidelines or requests for 
information to LEA*s long after the school year has begun. In fact, 
requests usually go out in the spring or near the end of the academic 
year when the State is in a very poor position to influence either the 
evaluation design or the kind of information collected on Title I projects. 
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As a result 3 the project managers only collect information readily avail- 
able to them. Furthermore 3 while most States merely collect whatever test 
data exist (as opposed to requiring uniform testing or the use of selected 
tests) 3 the test data are often not processed until late in the summer. 

The reports filed by LEA 1 s reach the State after decisions on project 
approval for the next year are made. 

As presently constructed, the annual State reports to OE provide little 
information on the extent to which projects are meeting stated objectives. 

In all too few instances do Title I project evaluation designs include com- 
parison or control groups. The information reported on achievement, while 
accounting for a significantly greater number of children than does the 
national survey (the PCI), contains nothing to indicate the characteristics 
of the reading projects for which data are reported or the characteristics 
of the pupils involved. States are asked to relate program costs to 
benefits, but the validity and reliability of the responses--where any are 
provided--are in question. 

The timing and the content of State reports minimize the extent to 
which the State affects local evaluation efforts and greatly reduces the 
utility of evaluative data in the State decision-making processes. 

What remains as the keystone of State evaluation activities is the 
variety of activities which we have called "monitor ing . " As has been 
pointed out, monitoring activities vary in content, structure, organiza- 
tion, and emphases across the States. No matter how haphazardly monitor- 
ing activities are carried out, however, they provide the direct opportunity 
for the State to influence local program development and to collect infor- 
mation used by State personnel in making decisions about project funding. 
Models for effective and efficient State program monitoring remain to be 
developed. Such models will have to take into account the variation which 
exists among the States in the way evaluation and monitoring activities 
are organized, the staff involved in such activities, and the resources 
available for their support. 
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D. Long-Term Strategy for State Evaluation Efforts 

This chapter has been concerned with State evaluation efforts viewed 
from both the State and the Federal perspective. The suggestions for a 
long-term strategy are intended to guide the Office of Education in its 
dealings with State Departments of Education in future years. The sugges- 
tions emphasize the need for OE to work with the States to develop and 
improve their capacity to evaluate programs. Because of the wide varia- 
tions in needs, resources and skills within State Departments of Education, 
OE should resist attempts to treat SEA's as if they were all of equal 
capacity. In other words, OE should be firm but flexible in its dealings 
with SEA 1 s . Until a great deal more is known about how SEA f s are organized 
and how they discharge their responsibilities, OE should attempt to work 
with some States to develop alternative models for State evaluation 
activities . 

1 . Recommendations for Improving Evaluation 
Activities Directed Toward Assisting State 
Planning and Management Efforts 

a. OE should entourage State evaluation activities which are 
directed toward the following high priority areas: 

(1) the improvement of monitoring within the State; 

(2) improved needs assessment within the State; 

(3) the development of project rating systems; 

(4) improved capacity to do comparative cost analyses; 

(5) the design and execution of program strategy 
evaluation. 

b. OE should encourage States to set up comprehensive evaluation 
systems which, at a minimum, include 

(1) the specification of measurable objectives for 
quality education within the State; 

(2) utilization of common output measures which 
measure progress toward those objectives; 

(3) provision for longitudinal, as well as cross- 
sectional evaluation; 

(4) the identification of "good" and "bad" practices 
with regard to compensatory education; 

(5) the widespread dissemination of evaluation 
findings; 
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(6) the means of identification and comparison 
of program costs, down to the campus level; 

(7) provision for the feedback of evaluation 
findings into State decision-making. 

c. OE should give highest evaluation priority to the improvement 
of State monitoring activities through the development, in conjunction 
with several States, of models for State monitoring systems. These models 
should probably contain a common core of information but should allow for 
variations in State organization and in the composition of the population 
of students and school systems in the State. The models should provide for 
systematic, comprehensive monitoring of projects, including such elements 
as : 



(1) purposeful assignment of staff to monitoring 
sites. The basis for assignment should include 
sending some of the same people to visit comparable 
projects ; 

(2) SEA preparation of pre-site visit profiles prior 
to the visit, based upon information available in 
the SEA (e.g., project application forms, prior 
evaluations, previous site visit reports); 

(3) use of a combination of open-ended and specific 
monitoring questions ; 

(4) training of monitors; 

(5) the use of monitoring teams (especially for 
large LEA’s) in which each member concentrates 
on a different area of program operation; 

(6) linking financial and program audits; 

(7) the filing of written site reports which 
include an overall assessment by the team 
of the LEA(s) or project(s) visited; 

(8) prompt feedback to the LEA visited; 

(9) adequate provision for follow-up of site 
visits ; 

(10) development of inexpensive, short-term measures 
of educational performance; 

(11) provision for annual review and revision of all 
or part of the monitoring system. 
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The development of several models will allow variation in such impor- 
tant areas as: (a) SEA staff used as monitors; (b) the mix of site visits, 

written materials, and statistical information used in the system; (c) the 
use of self-evaluation prior to a site visit; (d) the scheduling and length 
of site visits; (e) the number and type of progress reports filed by the 
LEA*s during the year. 

d. OE should spend a portion of its evaluation funds to help 
selected SEA’s develop models for State monitoring and to support improved 
State comprehensive evaluation. In addition, legislative authority should 
be sought for increased State Title I administrative funds, part or all of 
the increase to be earmarked for the support of monitoring and evaluation 
activities . 

2 . Recommendations to Improve State Evaluation 
Activities which Would Assist Local Planning 
and Management Efforts 

a. OE should encourage the States to foster cooperative local 
evaluation efforts which include the use of common evaluation designs and 
common output measures. 

b. OE should assist States to develop explicit instructions on 
the design of LEA project evaluations, to offer examples of appropriate 
techniques or instruments, recommend and/or locate consultants, and set 
guidelines for expenditures for such evaluations. 

c. OE should encourage States to seek out and publicize examples 
of both good projects and good local evaluation practices. 

d. OE should assist the States to develop regular and rapid 
techniques to feedback to LEA 1 s information about what other LEA's are 
doing elsewhere in the State. 

e. OE should work closely with the States in their efforts to 
get school districts to write performance objectives for Title I projects. 

3 . Recommendations to Improve State Evaluation 
Activities which Would Assist /oderal Planning 
and Evaluation Efforts 



a. OE should work with the States to restructure the format of 
the annual State reports to focus primarily on the reporting of the results 
of State monitoring activities. 

b. OE should encourage neighboring States or States with similar 
pupil populations or problems to collaborate in the development of similar 
evaluation and monitoring systems. 

c. OE staff should become familiar with the organization and 
operation of State evaluation and monitoring activities through continua- 
tion and strengthening of State Management Reviews, through augmented and 
improved Title I Program Reviews and through close coordination of evalua- 
tion efforts among OE programs (particularly Titles I, III, VII, VIII, and 
Vocational Education.) 
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E, Next Steps for Improving State Evaluation Efforts 

1. OPPE should allocate a part of the FY 1971 evaluation funds for 

the development of several models of comprehensive State monitoring 
systems, OPPE and DCE staffs should work closely with two or more States 
in the selection of a contractor and in the design, installation, and 
testing of the prototype monitoring systems, [Cost estimate for the 
development of each model: $25 , 000-$50 , 000 , ] Monitoring of these con- 

tracts should be the responsibility of DCE staff, 

2. DCE should develop a more systematic approach to its own 
monitoring of the State Departments of Education through improvements and 
refinements in its Program Review, Many of the elements specified in the 
recommendation (lc) of the previous section ought to be included in the 
DCE Program Review, When possible, the Program Reviews ought to dovetail 
with State Management Reviews, For example, the Program Review might be 
an extension of a State Management Review; in this case, at least one DCE 
staff member could participate in the SMR and then could be joined by other 
DCE staff to conduct a Title I Program Review. Also, as a part of the 
Program Review, DCE staff should participate, as observers, in SEA monitor- 
ing visits to LEA's. Revisions in the Program Review should be made in 
consultation with staff from BESE, the Division of State Agency Cooperation, 
and the HEW Audit Agency * 

3. DCE, in conjunction with the HEW audit staff, should develop 
training materials on Title I purposes, guidelines, and organization for 
use by State auditors. This is especially important since State auditors 
generally operate independently of Title I monitoring activities. 

4. OE should not make significant changes in the format of the State 
annual Title I reports for FY 1971, but DCE staff working closely with BESE 
staff and representatives from the SEA’s should begin to consider revisions 
to strengthen the State annual reports. The suggested models of comprehen- 
sive State monitoring systems, plus the insights and information obtained 
from the FY 1971 Program Reviews and the CPIR and PCI, should be important 
sources of information for revising the reporting format for FY 1972. 

5. DCE staff should develop and disseminate materials for State 
and/or LEA use in writing performance objectives, especially for program 
areas identified as of high priority (reading, early childhood education, 
etc.). As a background for this, DCE should attempt to discover the 
performance objectives now being used in various types of programs. 

6. One DCE staff member should be assigned to the development of the 
Management Appraisal System, the part of the Belmont system concerned with 
developing an approach and a system for evaluating and improving management 
in education agencies. The Texas Education Agency has just received a 
contract to begin work, in conjunction with several other States, on this 
system. 
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IV. EVALUATION OF EDUCATION PROGRAMS FOR THE 
DISADVANTAGED: THE LOCAL ROLE 

This study focuses on Title I evaluation and technical assistance at 
Federal and State levels. To round out the evaluation picture, this 
chapter discusses constraints on local evaluation efforts and suggests 
some strategies for strengthening the local role in Title I evaluation. 

A. Types of Evaluation and Evaluation-Related 
Activities Most Appropriate to LEA Efforts 

Urban Institute site visits to the State Education Departments in 
Massachusetts, New Jersey, Oregon, Pennsylvania, and Texas indicate that 
SEA staff members believe that LEA's vary widely in their ability and 
potential to use local evaluation efforts in the following tasks: 

-- To assist local program planning and program development; 

-- To improve local program operations; 

-- To assist State program planning and program management; 

-- To assist Federal program planning and evaluation. 

Information provided by State personnel in the five States visited by 
Urban Institute staff suggests that LEA's are best equipped to perform: JL / 

— Data collection on services provided, populations served, 
and costs of providing services; 

-- Local project evaluation (including reliance on outside 
evaluators and cooperative local evaluation efforts 
across LEA boundaries). 

Local districts would appear least able to perform: 

-- Program impact evaluation; 

-- Program strategy evaluation; 



1 / Of course, there is a wide range in the ability of LEA's and local 
projects to perform each of these types of activities. In general, 
larger sized LEA's are better equipped than medium sized and smaller 
ones . 
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-- Project rating; 2 J 

-- Monitoring; 

-- Comparative cost analysis. 

B. Constraints on Local Evaluation Efforts 

Urban Institute interviews with State agency staff in five States and 
data collected by the Great Cities Research Council 3/ suggest the following 
constraints on local evaluation efforts: 

1 . Financial - - lack of money to develop adequate evaluation designs , 
hire outside consultants, develop computer capability, etc . Since most 
projects spend approximately one to two percent of their Title I funds on 
evaluation, 4/ it might be desirable to increase the proportion of funds 
devoted to evaluation (at least in some LEA’s). Part of OE’s evaluation 
money could be used to support promising local project evaluations. The 
Office of Education should provide direction on the appropriate proportion 
of funds which should be allocated for local project evaluation and on the 
desirability of performing project and program monitoring and process 
evaluation. 

2. Polit ical -~ since most SEA*s are extremely sensitive on issues that 
might involve tl local autonomy, f> LEA 1 s customarily are not required to make 
appropriate changes in educational practices as indicated by evaluation 
results or monitoring^ activities . One gets the distinct impression from 
State officials that they are as wary and cautious in their handling of 
LEA*s as the Federal officials are in dealing with SEA’s. 5/ In addition, 
until very recently, public pressure for accountability in education has 



2/ Some LEA’s may be making progress here. For example, the New Jersey 
SEA and the 26 largest LEA’s in New Jersey currently are discussing 
a procedure whereby these school districts will turn back one percent 
of their Title I allocation to be used for cooperative, comparable 
evaluation studies. All 26 districts would use the same testing 
instruments (designed for longitudinal testing), and woul J collect 
socioeconomic data on students to allow for comparisons of achievement 
levels in various schools. 

3/ See John L. Hayman, Jr., et al . , "Title I in the Great City Schools: 

An Analysis of Evaluation Practices and Exemplary Projects," The 
Research Council of the Great City Schools, July 29, 1970. 

4V See February 7, 1968, Minutes of the Directors of Instructional 

Research in Large Cities . During FY 1968, approximately 11% of LEA 
Title I funds were used for administration (including evaluation) , 
operation and maintenance of plant, and fixed charges. The FY 1970 
Consolidated Program Information Report will include the first local 
estimates of LEA Title I evaluation expenditures in a nationally 
representative sample of LEA*s. 

5/ David K. Cohen and Tyll R. van Geel, "State Government and Poverty: 
Public Education in Massachusetts," Harvard University Institute of 
Politics, 1970. 
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had little impact on local evaluation activities. So long as OE provides 
no incentive to SEA 1 s to do real evaluation or to improve evaluation, and 
so long as States, in turn, provide no incentive for local evaluation, 
projects will continue to be funded with or without evaluation components 
and in most instances, irrespective of evaluation results. As van Geel 
has noted: 

"In terms of the evaluation product produced, ... the 
fact that emerges is that evaluation efforts have been 
politically safe but educationally meaningless. No 
one is offended, no names arc named, no invidious com- 
parisons are made. These evaluation reports affect 
the interest of no one since they hardly lend themselves 
to the decision-making process and the reallocation of 
resources . " ^6/ 

3 . Administrative - - lack of sufficient staff to design and implement 
comprehensive evaluations of numerous and varied Title I projects . While 
larger LEA’s have been more successful than many SEA’s in recruiting and 
retaining competent evaluation staffs (due to more attractive salaries and 
working conditions in large part), LEA 1 s also have had to contend with 
substantial staff turnover rates and conflicting demands on research staffs. 

(Constraints 1, 2, and 3 are in agreement with those found by the 
President’s Task Force on Urban Education.) 7/ 

4 . Methodological - - needs for assistance in establishing mechanisms 
for data collection and ^processing and in selecting and applying appropriate 
evaluation tools . Educational evaluation at the local level, as at the 
State level, is hampered by a lack of specificity in project and program 
objectives, inadequate or poorly defined output measures, and inability to 
describe process variables simply and inexpensively. 

5 . Scheduling - - the timing of evaluations with respect to decisions 
on the next round of project applications has been poor . There is no 
congruence between the cycle of evaluation and either program planning or 
decisions concerning funding. Annual reports for one school year are sent 
to SEA’s after decisions have been made concerning programs for the next 
school year. 

6. Lack of support from SEA --it has been observed that some SEA ’ s 
work more closely with those LEA’s receiving the largest proportion of 
Title I funds . In New Jersey, the largest 26 LEA’s are most involved in 
cooperative efforts with the SEA. And in Massachusetts considerable 



.6/ Tyll R. van Geel, ’’Evaluation and Federalism," Harvard University 
Graduate School of Education, 197C, p. 33. 

2 J See Congressional Record, January 19, 1970, pp. H9 ff . , and January 20, 
1970, pp. E21 ff. 
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attention is given to the five largest LEA's (which account for over 
50 percent of Title I funds). In these States, small sized and medium 
sized LEA's are not only less likely to possess evaluation expertise, but 
also are less likely to receive technical assistance in evaluation-related 
activities than their larger counterparts. 

7. Lack of support from ancillary structures -- many LEA's do not 
appear to have taken advantage of potential relationships with colleges 
and universities or nearby Regional Educational Laboratories . There is 
some evidence, however, that these structures have not always been 
responsive to LEA's that have requested evaluation assistance. In any 
event, LEA's could benefit from assistance on new evaluation techniques, 
establishing performance objectives, relating processes to objectives, 
etc. If they are unable to form the appropriate linkages to receive this 
assistance, or if the SEA is unable or reluctant to intervene and estab- 
lish the necessary mechanism, this would be a serious handicap to local 
evaluation efforts. 

8 . Lack of support from school administrations - -several Title I 
research directors and coordinators report great difficulty in obtaining 
clearance for evaluation activities from intermediate administrative 
echelons . At the Great Cities Research Council conference, 8 / Title I 
research directors and coordinators commented that, while in some of the 
large LEA's evaluation staffs have complete freedom to "tell it like it 
is," some staffs are pressured to report "good" results and others look 
for evidence of "failure" to report to decision-makers. 

9. Lack of support from classroom teachers -- several Title I research 
directors and coordinators report that growing teacher militancy has com- 
bined with a view of evaluation as an intrusion on regular classroom 
procedures (especially at the end of the school year) to prevent the col- 
lection of good evaluation data . 

10. Hostility and resistance to evaluat ion --it would appear that many 
LEA's share the resistance of SEA's to the notion of comparative assessment . 
Partly related to strong feelings on the issue of "local autonomy," 
undoubtedly this resistance also is based on fears about decisions that 
would be made as the result of the use of evaluation. 

11. The way in which decision-making takes place -- Title I not only 
calls for the establishment of a new program priority at the local level 
(i.e., special attention to disadvantaged children) but also for a new 
process priority at the local level--a process that attempts to be vastly 
more rational and rigorous than the way decision-making is now conducted 

in local school systems. 9/ Among the obstacles to reforming the decision- 
making process of a local school system are: a generally uncritical 



8/ The conference was held June 7-9, 1970, at Lake Placid, New York. 

9/ See Tyll R. van Geel, op. cit . , pp. 63-72, and David Rogers, 

110 Livingston Street: Politics and Bureaucracy in the New York City 

School System , 1968. 
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satisfaction with regard to the job the school system is doing; the 
organization of the system not so much as an education-program-producing 
machine as an employment system, 10 / where one of the unspoken rules of 
the game is that one avoids as much as possible hurting the interests of 
another person in the system; 1_1/ a definite anti-intellectual, anti- 
rational vein within the school department; an internal organizational 
structure that is not conducive to effective planning; and the isolation 
of the school system which not only fails to import large numbers of new 
and diverse kinds of people each year, but also adopts a defensive posture 
and isolates itself from forces within the community that would change it. 

In short, it may be that there is a fundamental incompatibility 
between the demands of evaluation and the vested interests of many public 
school systems as they operate today. 12 _/ What is evident is that there 
has been a clear absorption of all Federal and State attempts to improve 
the quality of evaluation. The techniques of absorption were many: "do 

the absolute minimum necessary to meet the legal requirements (which are 
minimal at best); simply delay in the execution of the specific recommenda- 
tions of the State; point-blank refusal to comply with some of the sugges- 
tions; and execute some of the suggestions so poorly that they need to be 
done all over again." 13/ Other delaying tactics are to argue at great 
length about the inappropriateness of the tests in relation to the objec- 
tives of a program and to confuse the issue as to what the objectives of 
the program are. While not all of these tactics are intentionally planned 
to postpone educational reform, nevertheless, the effect is very much the 
same . 



Unless and until these obstacles to reforming decision-making in the 
LEA are overcome, evaluation requirements will have little chance of 
bringing about real changes. 14/ For it is abundantly clear that unless 
a school system accepts the full implications of an evaluation requirement, 
it will simply do its evaluations in a pro forma manner in order to remain 
eligible for Title I money. 



10 / The Boston school system, for example, has traditionally been 

characterized by a high degree of in-breeding. Large numbers of 
persons schooled in Boston have returned to teach in the same system. 

11 / Since evaluation potentially involves a critical comment on how well 

people are doing their jobs, there is little sympathy for the activity. 

12 / Tyll R. van Geel, op. cit . , pp. 57-58. 

13 / Tyll R. van Geel, op , cit . , pp. 31-32. If it is thought that Boston 

might be atypical in the techniques used or in the nature and quality 

of its evaluation reports, van Geel notes that studies by New England 

Data Systems and the Massachusetts Advisory Council on Education show 
that the evaluation efforts elsewhere in New England, including 
Massachusetts, are if anything of lower quality than those carried out 
in Boston. These reports indicate that despite the legal requirements, 
as many as half of the LEA's have avoided evaluating their Title I 
programs, and that most of the local school systems that have evaluated 
their programs have compromised the whole concept of rigorous evaluation. 
14/ Ibid. , p. 63. 
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C. Conclusions and Recommendations Concerning 

the Local Role in Evaluation 

In spite of the identifiable constraints to evaluation at the local 
level, the climate appears favorable to substantial progress in many 
communities. The demand for evaluation from various educational funding 
sources, the increased participation of industry in education, the emphasis 
on dissemination and accountability, the thrust toward writing performance 
objectives, the gradual improvement in training programs in evaluation, and 
the increase in the number of empirically oriented members of lay boards 
and governing boards of education are seen by research directors and Title I 
coordinators as interrelating to create a more favorable atmosphere for 
increasing the LEA's participation in educational evaluation and for improv- 
ing the quality of the evaluation that is performed. 

With these factors in mind, four major recommendations are offered to 
increase and strengthen the local role in evaluation and evaluation-related 
activities : 

1. The kinds of evaluations that local school systems should be 
carrying out are neither applied nor basic research in the strict sense 
of those terms. Rather, evaluation at the local level should be manage- 
ment-oriented, with a stress on quality control. Local evaluation should 
make sure that the needs of students are being met, that programs are 
executed as planned, and that results are assessed. 15 / 

In particular, the Division of Compensatory Education should work with 
SEA’s and LEA’s to help develop new accounting or cost estimating systems 
to increase local evaluation capabilities and to provide data to meet SEA 
and Federal requirements. For example, in order to assist decisions on 
whether comparability is being achieved in the expenditure of local funds 
in Title I and non-Title I schools within a school district, data will be 
needed on services being delivered to individual schools and costs of such 
services. At the present time, only a relatively small number of school 
districts can provide expenditure data on a school-by-school basis. 

2. LEA’s should work together to establish cooperative mechanisms 
for more uniform, comparable, objective evaluation of the degree of effec- 
tiveness of alternative compensatory education strategies under varying 
circumstances. As indicated earlier, these cooperative efforts should 
provide for systematic examination of major program variations; collection 
of comparable data on costs, process, and outputs; and collection of com- 
parable follow-up data on the impact of the different approaches used. 

3. Since local Title I evaluation dollars are often wasted on 
unrelated, noncomparable evaluations of individual Title I projects, 16 / 
groups of LEA’s should organize to collect comparable output data for 



15 / Tyll R. van Geel, op . cit . , pp. 4-5. 

16 / At the conference of the Great Cities Research Council, Title I 

research directors and coordinators indicated a recognition of the 
unnecessary duplication of data gathering among various projects and 
the lack of a consistent data collection procedure. 
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those projects with similar objectives serving similar groups of children. 

This would assist project rating and monitoring activities, greatly 
strengthen the Project Descriptor effort within the Belmont system, and 
could provide the basis for subsequent program strategy evaluation. LEA f s 
and local school superintendents and school system staffs also could use 
such information to assist decision-making on school budgets and to reach 
decisions on the allocation of existing resources. 

The Office of Education should encourage cooperative Office of 
Education-State Educational Agency-Local Educational Agency and Office of 
Education-Local Educational Agency-Local Educational Agency efforts to 
introduce greater comparability into local evaluations of Title I projects 
with similar objectives. OE evaluation funds and technical assistance 
should be channeled to groups of cooperating States or groups of cooperating 
localities that share similar problems and are willing to do comparable 
compensatory education project evaluations from which all can learn. These 
cooperative efforts should provide for systematic examinations of major 
program variations; collection of comparable data on contexts, costs, and 
short-term and longer-term outputs. 

The Office of Education should attempt to limit unrelated evaluations 
of individual local projects to timely comparisons of project results with 
previously stated performance objectives or with baseline conditions, and 
should attempt to induce greater comparability in statements of local 
Title I project objectives, in measurements of the degree to which project 
objectives are attained, and in descriptions of the populations served and 
of the contexts in which the projects operate. 

The Office of Education and SEA's should encourage LEA 1 s to use common 
evaluation instruments and equal intervals between test administrations. 

The Belmont staff should take the lead in these efforts. 

4. States should encourage LEA's to make available to parents and 
community representatives measures of the distribution of Federal, State, 
and local funds to each school in a school district and periodic assess- 
ments of the standing of each school on several output measures. 17 / Since 
the Title I legislation requires the participation of citizen advisory 
groups in planning and evaluating' the Title I program, SEA's could require 
LEA's to provide such information in order for the advisory groups to dis- 
charge their responsibilities. To make these efforts meaningful, it will 
be necessary to collect social class data on the student body in each 
school on an annual basis, and perhaps on some other indicators of the 
student population and the neighborhood in which each school is located. 

LEA's should develop ways for teachers, parents, administrators, and 
students to work together to develop project goals and criteria of project 
success. LEA ' s also can provide important inputs te a system for Federal/ 
State monitoring of Title I projects by organizing local advisory committees 
of community residents to provide regular assessments of Title I projects. 18 / 



17 / At the conference of the Great Cities Research Council, several Title I 
research directors- and coordinators favored assigning researchers to 
work directly with parents and teachers. 

18 / At the above-mentioned conference, many of the persons assembled sug- 
gested the use of in-class observation by parents as part of the 
evaluation process. 
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I). Other Research on the Local Role in Evaluation 

Research efforts of the Great Cities Research Council should add to 
the information presented and sharpen the picture of the local role in 
evaluation and evaluation-related activities. 

The objectives of the research by the Great Cities Research Council 
are as follows: 19 / 



(a) 


To produce examples of "good" Title I programs from 
the 20 members of the Council. 20/ 


(b) 


To identify factors related to the success or failure 
of Title I programs. 


(c) 


To suggest ways to improve Title I evaluations. 


(d) 


To broaden the scope of evaluation efforts. 


Data have been collected from interviews with Title I research 
directors and coordinators and at a conference of research directors 
coordinators, curriculum specialists, teachers, and administrators. 
Analysis of these data produced recommendations on: 


(a) 


The assignment of organizational responsibilities 
for evaluation. 


(b) 


Resources needed and available to support local 
evaluation efforts. 


(c) 


The impact of local evaluation activities on 
policies and practices affecting the education 
of disadvantaged children. 



While the conclusions drawn from the Great Cities interviews have 
been presented in their entirety in the Great Cities Research Council* s 
report, 22 / we focus here on the six principal findings and recommenda- 
tions that most specifically pertain to local evaluation efforts: 



19 / These objectives are taken from a Great Cities Research Council 
document . 

20 / An additional purpose is to identify successful evaluations of Title I 
projects done by locals or outside contractors. 

21 / The conference was divided into four sessions designed to contribute 
information about Title I project design and/or evaluation. The 
sessions were titled: Problems, solutions, and fiascoes; problems in 

evaluating Title I projects; developing Title I evaluations from case 
studies; and force field analysis. 

22 / See John L. Hayman, Jr., et aL, op. cit . 
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(a) Study is needed to develop methods for continuous 
monitoring of projects and for other aspects of 
process (or formative) evaluation in a non- 
threatening and economically feasible way. Con- 
sideration should be given to varying evaluation 
methods according to project size, complexity, and 
developmental state . 

Comment . While such efforts undoubtedly would be 
methodologically valuable, they would be extremely 
expensive and time consuming. We have recommended 
that the Office of Education should give highest 
priority to short-term, relatively inexpensive com- 
parisons of project outcomes with previously stated 
performance objectives or baseline conditions. 

(b) The effects of many programs carry on over a period 
of years. A concerted effort is needed to develop 
methods of assessing project effects over a longer 
period of time. 

Comment . We recommend that SEA's be urged to use money 
provided under Section 402 of ESEA Amendments for this 
purpose. In addition, we have indicated elsewhere that 
the Project Descriptor effort within the Belmont system 
has begun to collect comparable data on local projects 
in a number of Federal education programs (including 
Title I). The effort provides statistical and narrative 
data on the context in which local projects operate, the 
organization and budget of the local projects, and the 
changes achieved by the projects. The Project Descriptor 
effort could provide the basis for subsequent longitudinal, 
program strategy evaluation at the Federal level, and also 
could assist LEA f s in project rating activities. 

(c) Some important effects of current Title I projects 
are unanticipated, hence unmeasured. A systematic 
study of those effects identified as unanticipated 
would help evaluators of future projects determine 
in advance what effects might be expected. 

(d) There is currently national recognition of the need 
for precise behavioral objectives for both project 
planning and evaluation. Efforts at ^preparing behav- 
ioral objectives need to be more widely publicized. 

(e) Current evaluation practices seem to rely too heavily 
on standardized achievement tests. Consideration 
should be given to the wider range of variables that 
need to be measured and to the kind of instrumentation 
needed . 
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Comment . Our discussion of research instruments to 
measure progress toward Title I objectives and require- 
ments for further instrument development (see Chapter V) 
is in substantial agreement with each of these findings. 

However, it appears to us that the current state-of-the- 
art in instrument development and related tasks is 
somewhat more advanced than recommendations (c) , (d) , 
and (e) might lead one to believe. Table 1 of Chapter V 
represents an attempt to identify types of information 
needed to measure progress toward Title I objectives 
(including outcomes that might have been unanticipated) 
and a wide range of specific research instruments now 
available to obtain this information. 

(f) With the current financial problems faced by most 
large school systems, the temptations are great to 
reduce the expenditures for evaluation. A minimum 
of five percent of Title I funds is suggested for 
evaluation, with additional funds available for 
those projects requiring more costly evaluations. 

Comment . It is our judgment that the cost of allocating 
a minimum of five percent of Title I funds to local 
project evaluation (more than $50 million per year) would 
be prohibitive. While we have recommended that the Office 
of Education should provide direction on the appropriate 
proportion of funds which should be allocated to local 
project evaluation, we have stressed that the emphasis be 
placed on monitoring and related activities. Done on a 
sampling basis, monitoring could provide timely informa- 
tion on the strengths and weaknesses of local projects 
and could collect information on the experience of school 
districts with similar problems or projects. 

Finally, an examination of the Great Cities interview data reveals 
that Title I research directors and coordinators identify the same projects 
and evaluations as successful and hold very similar opinions on criteria 
for successful local evaluations and problems of evaluation at the local 
level. 

The major problems identified in the development of Title I project 
designs were training research personnel, decision-making, developing 
procedures for measuring unintended effects, and not being able to carry 
out process evaluation. Major problems identified in the evaluation of 
Title I projects were frequently changing objectives; a lack of evaluation 
of the impact of projects .on parents, teachers, and communities; insuffi- 
cient attention to such output measures as organizational change, community 
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attitudes, and changes in teacher behavior, and a shortage of local con- 
sultants. A content analysis of the remarks made at the conference 
session on problems, solutions, and fiascoes reveals that there were many 
more examples of problems than solutions provided. 23 / 



23 / Unfortunately, there is no way of assessing how many conference 

participants held particular opinions. No statistical accounting 
was performed for any of the conference findings. Conference find- 
ings cited in this chapter rest on the judgments of Great Cities 
Research Council staff members. 
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V. TITLE I EVALUATION: IMPLICATIONS OF EXISTING METHODOLOGY, 

INSTRUMENTS AVAILABLE TO MEASURE PROGRESS TOWARD TITLE I 
OBJECTIVES, AND REQUIREMENTS FOR FURTHER 
INSTRUMENT DEVELOPMENT 



This chapter examines existing evaluation methodology, assesses its 
limitations, identifies requirements for more effective evaluation, and 
explores ways that Federal, State, and local energies might be directed 
toward this end. The discussion includes the identification of research 
instruments available to measure progress toward Title I objectives. 

A. Implications of Existing Methodology 

Until now, most evaluations of federally aided compensatory edu- 
cation programs have failed to meet even minimal requirements. The best 
of the Title I project evaluations done to date usually are little more 
than educational audits designed to meet annual deadlines set by the 
Office of Education and State Departments of Education. They do not begin 
to explain the processes that bring about the results that have been 
observed. 

More specifically, educational researchers report a host of evaluative 
problems appearing with considerable frequency in programs of compensatory 
education. Some of the more common situations, with obvious implications 
for evaluation methodology are as follows: 

1. While it generally is assumed that evaluative studies are con- 
ducted to guide decisions concerning the continuation, expansion, redirection, 
or elimination of educational projects, an examination of many educational 
evaluations reveals that decision-making and evaluation are carried out 

(and even perceived) as totally unrelated activities. 

2. Evaluation generally has been conceived as a separate endeavor 
from program development, and the thrust of much educational evaluation has 
been to take a total program and judge it as either worthy or unworthy. 

This approach, which has limited usefulness because it fails to relate 
treatment to events, contrasts with efforts to find projects that are 
working better than others, identify why , and improve and replicate them 
in other settings. The answer to this question "why" requires an analysis 
of such factors as: (1) the attributes of the project which make it more 
or less successful; (2) the population exposed to the project, in terms of 
which subgroups are reached and which affected; (3) the situational context 
within which the project operates; and (4) the different effects produced 
by the project, including special attention to any important positive or 
negative side effects. 
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3. The pupil population being observed differs. In some States, all 
pupils in a classroom, grade or school are included in the evaluation; in 
others (such as Ohio and New Jersey), individual pupils are the basis of 
the evaluation. 

4. While the heterogeneity of groups presents a natural situation for 
multivariate analysis, most school districts are doing little of this and 
are struggling with setting down objectives in operational terms. It can 
be noted here that lack of local expertise in evaluation does not seem to 
have resulted in large numbers of outside consultants being asked to evaluate 
local projects. For financial and possibly other reasons, local districts 
usually do their own evaluations. 

5. In many school districts, the same pupils are involved in several 
projects. This raises the question of whether or not it is possible to 
sort out the (interactive) effects of these projects. 

6. In numerous evaluations of the "pre-test, post test" variety 
difficulties in interpretation have resulted because of differences in 
tests used, students used, administration procedures, and statistical 
treatments . 

In an earlier Urban Institute paper four evaluative problems more 
directly related to research instrumentation were identified: 1/ 

1. Local and State personnel are unclear about the Office of Education's 
priorities for Title I and other federally aided compensatory education 
programs, 2 / resulting in considerable confusion over which "output" 
measures should be studied. The confusion stems, in part, from disagree- 
ments over the relative importance of Title I 1 s several purposes. 

2. Some Title I projects may be unevaluable in useful terms. In any 
event, Title I projects as funded and approved seldom add up to a well- 
coordinated program in any school or school system. As a result, Title I 
projects do not resemble the kind of "package" the professional evaluator 
thinks about when he thinks of educational evaluation. 3 / 

3. In selecting evaluation criteria for determining project or 
program success or failure, evaluators have concentrated on children 
directly participating in the project or program in question, and have 
largely ignored the effects of those not in the direct focus of the program. 



1 / See Joseph S. Wholey, ejt al. , "Title I Evaluation and Technical 

Assistance: Report to the Title I Task Force," Urban Institute Working 
Paper 954-21, August 7, 1970, Appendix G, pages G-6 to G-8 and G-13 to G-15. 

2 / This was reported at the April 27, 1970 Conference on Technical 
Assistance sponsored by the Office of Education and Institute for 
Services to Education, and during meetings with members of State Depart- 
ments of Education. 

3 / See Henry S. Dyer, "Evaluating Educational Programs: A Symposium," 

in The Urban Review , February 1969, page 11. 
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4. Evaluators have tended to ignore the relative importance of two or 
more criteria when several criteria for determining project or program 
success or failure may be sensible. The problem of weighting criteria must 
be addressed if an overall evaluation is to be made. 

B. Instruments to Measure Progress Toward Title I Objectives 

With the above problems of research instrumentation in mind, this 
section identifies existing research instruments that can be used to measure 
the extent of progress toward Title I f s many objectives and discusses 
requirements for further instrument development. As increased emphasis is 
devoted to setting project objectives in measurable terms, and to encouraging 
evaluation against these objectives, the identification of these instruments 
and related information (e.g., date of publication, publisher or developer, 
time for test administration, cost of administration, etc.) should be of 
great value to LEA's, SEA's, and the Office of Education in both their 
evaluation and technical assistance efforts. 

The Title I legislation and amendments, the literally thousands of 
Title I project proposals each year, and members of five State Departments 
of Education have clearly articulated and implicitly suggested a great 
number of varied (and frequently changing) objectives. While there are 
numerous ways to categorize these objectives, 4/one convenient way is 
according to target population. 5/ .• ' . 

1. Elementary School and High Scho # ol Students (and Dropouts) 6/ 
a. To improve cognitive development 

(1) Language skills 

(2) Communication skills 

(3) Computational skills 

(4) Conceptual skills 

(5) Perceptual skills 



4/ Many of the sub-objectives in categories 1, 2, and 3 are suggested in 
Fry Consultants, Inc., Improving Coordination of Education Programs 
for the Disadvantaged , June 12, 1970. 

5/ This list of Title I objectives and sub-objectives is not meant to be 
exhaustive. Rather, it represents an attempt to identify the most 
common Title I objectives to which specific research instruments can 
be attached later. 

6/ The reader is cautioned that this categorization excludes migrant 
children and neglected and delinquent children served under the 
Title I program. Of course, many more Title I projects are aimed 
at elementary than high school students. 
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